-
Notifications
You must be signed in to change notification settings - Fork 36
Meeting Notes
-
As of two days ago, DCSOps is in "no new incididents can be created" mode.
-
We'll make DCSOps fully read-only on Fri Aug. 20th.
-
We'll deliever the data on or before Fri Aug. 27th.
Raw SQL dump (PostgreSQL), plus CSV files of incident and assignment data (OTS to be guided by Jim McGowan in determining CSV columns).
-
We'll shut down DCSOps immediately after delivering the data.
-
Forward DCSOps.org to https://volunteerconnection.redcross.org/ .
Darlene will follow up with Jason Shapiro regarding transfer of "dcsops.org" domain name to Red Cross (as per discussion in March 8th meeting, see below). Until the transfer happens, OTS will forward the domain to Volunteer Connection. If there's any delay in the transfer, OTS will probably switch the domain to our own registrar (gandi.net), because it's cheaper than the current registrar (dnsimple.com), and we'd maintain the domain there for up to a year if needed.
-
Darlene is following up regarding the past invoices.
Those payments might be on a 45-day instead of 30-day cycle, in which case it would make sense that they haven't arrived yet, since they got approved on June 30th or so and it's only early August yet.
-
All FY21 work completed. Invoice sent (20210308-1).
-
Steve Meissner is still the Zendesk Admin; should be Clara.
ACTION: Clara to take care of this.
-
Updated documents.
Jim has updated the What's New (and What's Fixed) in DCSOps document.
ACTION: Clara will decide how this document should be shared and how the regional questions the document generates should be handled.
The Technical Support buttons on each region's homepage now point to this document.
ACTION: Jim will convert both of these documents to PDFs hosted directly in DCSOps, once they are stabilized, and will update all links and leave forwarding pointers from the old docs in Basecamp.
-
Transferring "dcsops.org" domain (and maybe "arcbadata.org") to ARC.
ACTION: Darlene will ask Jason Shapiro about ARC taking one or both of those domains over.
ACTION: OTS will find out whether "arcbadata.org" is even needed anymore.
-
Closing out the Postmark account now that DCSOps uses Sendgrid.
ACTION: OTS to take care of this.
-
List of DCSOps->RCResponse migration groups.
ACTION: Clara will make sure Jim gets this.
-
RC Care work invoiced under wrong PO line.
Just got PO CP2395-3 from Pamela Amerson today, right after OTS submitted the invoice under CP2395-2, dang :-). Should be under the new number.
ACTION: Darlene to void the current 20210308-2 invoice, then Karl to update invoice and resubmit under PO CP2395-3.
-
What PO to use for the SSO work?
Use CP2395-2 for SSO.
-
What PO for upcoming DCSOps hosting invoice(s)?
These go under CP2395-2 as always.
OTS will invoice Jan-Feb in one invoice, then starting with March invoice switch to invoicing monthly,
-
Discussed the "210303" outstanding invoices spreadsheet.
-
Invoices 20160902-1 and 20160902-2 are paid.
Karl confirmed this (via 2017-05 bank statement) right after the call. These invoices can be removed from the list.
-
OTS doesn't have countersigned copy of RC Care / SSO amendment.
We do have correspondence with Tish Whitaker, so it's not really a problem, but it would be nice to get the fully executed copy.
Attendees: Darlene Johnson, Clara Walker, Jim McGowan
Jim reviewed the completed FY21 work:
- Rails updated to version 5.2.4.5. (#157)
- Application code moved to the redcross/dcsops repository on GitHub.
- By-region admin access controls implemented. (#166)
- Built-in reporting now available replacing Sisense/Periscope dependency. (#301)
- Incidents
- Shifts
- People
- Transactional e-mail provider changed from Postmark to Twilio SendGrid. (#213)
- IIR template updated to match National standards. (#260)
- Regional Admin can now change an incident report’s date. (#262)
- When calculating mileage, only ‘Team Lead’ and ‘Responders’ are summed. (#259)
- Dispatch New Incident script modified and Notification Source pop-up menu added. (#263)
- Time spent responding for each responder can be calculated however, we can no longer export directly to Volunteer Connection. Instead, hours can be exported to CSV. (#215)
- Application crash when attempting to add a New Price List Item fixed. (#264)
Darlene will cross-check the list above with the FY21 statement of work.
All work has been summarized (in draft form) here for Regional POCs: DRAFT: What's New (and What's Fixed) in DCSOps.
Jim discussed other DCSOps issues that have been addressed over the last week:
- New Jersey users are stating that they are:
- Regularly being logged out of DCSOps and having to log back in
- Experiencing slow performance across the application. We believe these issues have been fully resolved.
- We identified an issue that may have prohibited incident reports from being created when DirectLine (which services Cascades, Gold Country, Northern California Coastal) sends incomplete address information to DCSOps. We have added 'Region' as a default response territory when no other response territory can be identified.
- The Illinois Volunteer Services team inadvertently removed administrative privileges (like Query Tool) which broke the daily export of volunteer data. Privileges have been restored and exports are now occurring as scheduled.
- The Minnesota-Dakotas dashboard was modified to reflect newly requested shifts (Operations, Senior DPM). Shifts weren’t visible because Service Details were not added when positions were created in Volunteer Connection
- Nebraska and Iowa changed a number of Volunteer Connection position names causing DCSOps schedules to break. Positions have been ‘re-mapped’ in DCSOps.
Darlene informed that Open Tech Strategies can now invoice for RC Care work.
-
Darlene to set up new FCC for RC Cares work.
Darlene's got the FCC now, just needs to circle back with Tish and then set everything up in Coupa. Once that's done OTS can send in the invoice for recent RC Care work.
-
FY21 work continuing on schedule.
Ruby-on-Rails production upgrade scheduled for Saturday.
-
Karl to send Tier 1 Support description to Darlene.
Has a draft of it from Jim; just need to tweak a bit and send along.
-
Upcoming ITIL initiative: how will it affect DCSOps?
"Information Technology Infrastructure Library" (ITIL) methodology initiative coming from senior leadership, across all of ARC. DCSOps is not one of the big enterprise-wide projects that this really affects, but still there might be some process changes needed for DCSOps. Darlene is still learning and will let us know.
-
Big meeting with Andy et al re outstanding invoices is tomorrow.
Darlene will let Karl know about outcome.
Darlene, Karl
Karl called Darlene briefly, to confirm to her that we would make the end-of-the-month deadline for completing all FY21 work.
Frank, Jim, Karl
-
The big three remaining FY21 SOW items:
-
Various much smaller remaining FY21 SOW items:
- Modify the IIR template (#260)
- Add to Regional Config and Regional Admin capabilities the ability to change an incident report’s date. (#262)
- Change the transactional e-mail provider from Postmark to SendGrid. (#213)
- When calculating mileage, sum for only ‘Team Lead’ and ‘Responders’ (#259)
- In the Dispatch New Incident script, modify the wording of two 'questions' and add a Notification Source pop-up menu. (#263)
- Calculate time spent responding for each responder and export the result to Volunteer Connection. (#215)
- Fix application crash when attempting to add a New Price List Item. (#264)
- Move application code to the
redcross/dcsopsrepository on GitHub.
That second list is collectively not a lot of work; all of it together is still less work than any of single item from the top three. But even any of the top three aren't very hard, at this point. So, we're going to complete everything by the end of the month. Jim will project-manage to the extent it's needed, keeping Clara in the loop and having her make decisions that NHQ should make.
Karl will let Darlene know that we're on track to complete by the end of the month.
Note that although #257 "data access for 3rd party call centers" is done per spec, it would be good to add a login portal for non-public data, and we will do that after the above stuff is done.
Darlene, Karl
Discussion about whether we will complete by end of month as per scheduled date. Karl was worried we might not, said he would go over the work with Frank and Jim, do some planning, and get back to her.
Darlene, Clara
-
Darlene has the new FCC number for the RC Care work now.
She will set up the new PO with Trish. Karl will hold off on invoicing until Darlene gives the green light.
-
Confirmed that Jim has indeed cleared up the DCSOps support backlog.
-
Clara is still working on getting a Volunteer Connection account for Jim.
-
Karl still to send "Tier 1 support" writeup for Darlene.
-
Karl still to submit SSO invoice and DCSOps hosting invoice.
-
Andy has scheduled a Feb. 23rd meeting re past invoices.
Senior VPs and DDEs and regional directors attending. Darlene will send Karl the spreadsheet she and Andy are working from, so Karl can double-check that everything's up-to-date.
Darlene, Clara, Rupa Hicks, Peter Cartwright, Catherine Melsheimer, Alayne Chapman, Jayme Snowden
-
Demonstration of DCSOps -> RC Care linking.
Darlene recorded this part of the session. FWIW we used RC Care EV-0011963.
Note: The DCSOps interface still refers to "CAS" in some places where it should say "RC Care". This is a bug that Clara and Frank and Karl have discussed, and OTS will fix it -- getting the interface right is, ahem, within the scope of our original SOW.
After Clara did the demonstration, everyone but Darlene, Clara, and Karl hopped off the call.
-
Zendesk tickets: OTS will take over ticket management from Clara.
Karl will send Darlene a writeup of how providing Tier 1 support wasn't in our original maintenance contract, and include an estimate of how much more it will cost for OTS to supply it.
Clara says Jim's VC account is close to ready. Clara will keep us apprised this week. Clara will also ask about getting RC Care access for Jim (even read-only would be enough).
-
Karl will send separate SSO invoice and DCSOps hosting invoice now.
-
Wait on sending RC Care invoice until Tish gives the thumbs-up.
-
Andy has a meeting scheduled in the next week or two re past work.
-
Summary of Zendesk support tickets
(just copied from Jim's summary in public Basecamp)
19 Open tickets (4 ARC, 12 OTS, 3 still unassigned)
-
Reviewing today
- #243 DCSOPS - Lisa McGee (New Jersey) (Oct 23, 2020, Open)
- #205 Please ensure that Mark Hassmiller and Matt Aruta can look at the "clients" tab of the DCSOps data. This is necessary for our DCSOMR - Santana, Bedzaida (Sep 10, 2020, Open)
- #173 DCSOps incident confusion - Rowe, Don (Aug 30, 2020, Open)
-
Require Frank's review
-
Requires direction from ARC
- #257 ID and MT Duty Officer Hours Not Transferring to VCN - Catherine Rawsthorne (Idaho and Montana) (Nov 16, 2020, Open)
-
Awaiting reply from user
-
Requires special effort for individual user
-
-
Karl checked to make sure Zendesk #257 is not part of FY21 work (it's not).
-
Status of VC account for Jim McGowan.
Clara will ping VC team on Tuesday or Wednesday if no account by then.
-
Invoice for RC Care and SSO switchover soon.
Karl sent in the signed contract today. We can invoice for the RC Care work and SSO work as soon as that's processed (technically, we could invoice for SSO now, since Darlene says it fits within the current project's budget, but for OTS it'll be easier to submit both invoices at the same time).
-
Payment status of FY21 invoices.
Darlene will discuss with Trish, then all of us will meet tomorrow afternoon. Karl has sent Darlene an email summarizing all the remittance addresses we've had. Since both our Chicago and New York addresses changed in Q4 2020, there might be some confusion in Coupa.
Darlene, Clara, Karl
-
RC Care changes are now all deployed to production.
-
Jim McGowan is now consulting (through OTS) and working on Zendesk tickets.
Clara has a call scheduled for Wednesday with Sara Rouse (sp?) to work out VC access for Jim.
Jim knows that these three tickets are highest priority:
- Zendesk #276 ("Recruitment Messages")
- Zendesk #280 ("GNY Assignment Messages")
- Zendesk #282 ("URGENT MISSING CASES")
After those three, he'll move on to the rest of the open OTS-assigned tickets, and then to all other open tickets. Regarding the above three, Jim said this today:
"I've looked at all three issues and will need DCSOps and Volunteer Connection access to troubleshoot. All look resolvable. #276 really needs to be split into three separate issues. #280 is likely an issue with the Responder Console. #282 is a DirectLine/user error issue. Once I get application access, I'll more throughly document the issues."
-
Wait for Darlene's word before invoicing for RC Care and SSO switchover.
-
Feb. 28th wrap-up date for FY21 work is still good.
If that's going to change, OTS should let Darlene know by Feb. 8th.
-
We've extended regular meetings through Monday March 1st.
The March 1st meeting will be a wrap-up meeting, since everything should be done before Feb. 28th.
-
Check-in on outstanding invoices.
Andy sent an email to the senior VP of DCS, and DDE, and they have a meeting scheduled.
The regular meeting was cancelled today due to a scheduling conflict. However, Frank and Karl talked, and Frank said the RC Care changes are all working in development now, and that he will push to production tomorrow morning. He also notes that at least NY has been entering RC Care EV numbers already.
Darlene, Clara, Karl
(Meeting moved a half-hour earlier today to accomodate a scheduling request from Karl.)
-
SSO Server switchover is done.
-
CAS -> RC Care transition.
In progress. Darlene notes that meeting with senior leadership is in mid-January (she wasn't sure of exact date) and that she'll be reporting on this then. Ideally it would just be done by then.
-
Zendesk tickets.
Clara is consolidating notification issues in Zendesk #276. There are really two types of notification issue right now:
-
Users are receiving recruitment messages but the system is not recording those users' responses to those notifications.
-
When a new incident is created, specifically in the NJ region, sometimes everyone on the distribution list gets the notification (as they should) and sometimes no one gets it. (But note it's not the case that some people get a given notification and some don't.) There is a user who serves as a primary POC in reporting these things. So one way to work on this would be to tell the region we're about to start testing, and include "TEST" in the fake incident titles, and then have that POC relay information.
TODO: Karl will discuss with Frank the next steps on Zendesk tickets #276, #280, and #282 and send an email update to Clara and Darlene in the morning.
-
-
We discussed the ordering of the remaining of FY21 items.
RC Care, then staging the Rails 5 upgrade (and deploying it once tested), then implementing the authentication part of issue #257, then reporting dashboards (#301), then issue #166. Darlene may want a finer granularity look at the ordering/priority of some of the other remaining FY21, and if so will follow up by email.
Frank, Karl
-
CAS -> RC Care transition.
In progress; should be done this week.
-
Zendesk tickets.
Frank handled Zendesk ticket #281, since was related to code we wrote and he knew what to do.
Clara is consolidating the notification problems. Frank has made several attempts at solving them, but it's frustratingly slow because he relies on users performing regular usage to generate new data for him to work with. He can't just test whether a change worked because the nature of the problem is that a large group of people do get a successful notification except for one person who should get one but doesn't.
Regarding the missing-cases issues: Some incidents are generated by outside sources (this is by sending emails or making POST requests into DCSOps, though Karl can't remember off the top of his head which it is). Zendesk ticket #282 is the most recent instance of this. It's not yet assigned to OTS, but we can anticipate that it will be.
Biggest ongoing problem is turnaround delay: Frank sends a question, but then moves on to other stuff while waiting for the answer because it would be inefficient to sit idle.
-
Remainder of FY21 items.
Main thing is staging the Rails 5 upgrade, implementing the issue #257 authentication (if MN stays on), and the reporting dashboards (#301). Issue #166 will come last.
Darlene, Clara, Jason, Karl
-
#257 (call-center dashboards)
Status: Public call-center page is now deployed, similarly to how Periscope did it, except better because it's always up-to-date now instead of being periodically screen-scraped. Working on authenticated login next. Clara will let Joe Reinnaman know (or perhaps has already done so). Frank will work with Joe to coordinate them switching over to the authenticated version as soon as it's ready.
-
SSO server switchover.
Will happen on or around Christmas Day.
-
RC Care
Next after SSO server. Small to medium, Frank estimates. It comes down to whether the Ruby Salesforce plugin just works out of the box, or whether we have to do some fiddling.
Karl still needs to send Darlene the paragraph description(s) of this and the SSO work.
-
Karl to send executed FY20 SOW, just for ARC bookkeeping.
-
Jason will contact Ivan Chou et al about "dcsops.org" domain transfer.
(As discussed in 2020-11-16 meeting.)
-
We'll use Feb. 28 as the closeout date for the CR for SSO and RC Care work.
-
Zendesk support tickets.
-
Running themes.
-
Product design issues. E.g., once we started sorting response territories within a region alphabetically, but then that messed up the thing whereby the first listed territory was automatically made into someone's default. So OTS is essentially doing two pieces of work per ticket: what is DCSOps doing, and what should it be doing. This furthermore has a delaying effect in that Frank then doesn't go to spend time on tickets until he has enough time to spend. And we used to get clear decisions about what not to solve, whereas now everything trickles up.
-
Multiple different issues per ticket.
-
Would help to do more summarizing and preparing for Frank. Right now he essentially has to read over the entirety of every ticket to figure out what the core issue is.
-
-
Review of open tickets assigned to OTS, with priorities listed for the first several, to help us know where to focus first.
-
#277:
Notification Texts and EmailsDo this first, if and only if it's fast to debug given the latest information in the ticket. Otherwise, put it fifth or so.
An escalation of already-reported notification issues.
-
#257:
ID and MT Duty Officer Hours Not Transferring to VCNDo this second (but note it may be related to #274 and/or #276).
Possibly complex, likely to require investigation/debugging in the DCSOps<->VC connection.
-
#276
FW: DCSOPS Issue- Recruitment MessagesDo this third (but note it may be related to #257 and/or #274).
(Note: not actually assigned to OTS yet.)
-
#274:
Notification Texts and EmailsDo this fourth (but note it may be related to #257 and/or #276).
An escalation of already-reported notification issues.
-
#216:
DCSOps notificationThis is fith or sixth, after the above.
Pamela Marquez is not receiving text notifications (that others are receiving). Her phone number is now in the ticket, so we can look in the logs for it.
-
#258:
ReportingPossible solved or at least alleviated by #301, which we'll be working on in early 2021.
-
#270:
DAT scheduling pagePossible addressed on Dec 1st, by work on #243?
-
#263:
Impersonation problemIssue might actually be DCSOps preventing people from signing up in the middle of a shift? If so, do we need to talk to Clara to figure out whether that's a desirable restriction?
-
#255:
Flex TimeNot clear to me what the desired behavior should be here.
-
#231:
DCSOPS > CAS Event Number button missing from all casesI think this might just be a manifestation of the CAS / RC Care transition?
-
#243:
DCSOPSA lot of different issues seem to be mixed together in this ticket. Some of them Clara is/was able to handle. The alphabetical sorting issue is an OTS thing, but I think we've done it already? I couldn't find the issue, but I recall reading in a previous Zendesk ticket that alphabetical sorting had been done somewhere. (Also, if we are implementing alphabetical sorting, we should see if we're accidentally solving GitHub issue #60, even though that one was closed three years ago as WONTFIX.) The most recent thing in the ticket is Frank saying "I can say that at least now, the shift territories are sorted by name. As for the missing data, maybe we should open a separate ticket with the details only dealing with that." Agreed, but we should make clear who owns that. If we expect Clara to do that, let's say so. -
#205:
Please ensure that Mark Hassmiller and Matt Aruta can look at the "clients" tab of the DCSOps data. This is necessary for our DCSOMR.Awaiting response from Lisa McGee to our request for more information, but Frank might know enough to solve this already. -
#173:
DCSOps incident confusionSerious problem with DirectLine and incident numbering. Possibly also some email notification issues revealed here (not sure if those are related to the incident numbering problems).
-
-
Darlene, Clara, Heather, Jason, Karl
-
#257 (call-center dashboards)
Going well; Karl and Frank had a brief check-in about it today.
-
DCSOps OAuth SSO configuration
We are waiting for Dale Siar to let us know that
sso.redcross.orgis ready to handle OAuth requests fromarcdata-staging.herokuapp.com(see details in previous meeting's notes). -
RC Care
This is next after #257 is done.
Darlene is still finding out whether invoicing for the RC Care work will need to be done as an amendment; will let us know.
-
Zendesk Support tickets
Karl will do a review of all tickets assigned to OTS and see if any of them urgently need attention.
-
OTS to send DCSOps November hosting and maintenance invoice.
-
Confirmed overall order of near-term development.
Now #257, then RC Care, then Ruby-on-Rails upgrade, then #301. (And SSO reconfiguration and #173 will each be done whenever OTS gets the needed information.)
-
Older invoices.
Andy continues to talk to DCS Finance about pending Smoke Alarm invoices and DCSOps pre-FY21 hosting and maintenance invoices.
-
We will meet next week (Dec 21st), but not the week after (Dec 28th).
Clara and Karl mutually confirmed that they are both working through the end of the month, though, so if there are things they need to coordinate in the second half of December they can do so.
Attendees: Frank, Darlene, Peter
Mainly just walking through the SalesForce API that Peter emailed, as well as passing credentials.
Notes in short form:
-
The Account is Created, and is read only
- The account is also api only, so no need to put credentials in opass. They will just live in the heroku settings variables
-
The App is created in SalesForce
-
There are most likely only 4 calls necessary to get the Object ID
-
The url to construct is:
-
Credentials passed to frank and will be encoded in heroku variables.
-
Next steps are to get an RC Cares event from a region that's moved over, and test in DCSOps. Frank to reach out to someone in Group B later this week.
Darlene Johnson, Shana Jerman, Clara Walker, Dale Siar, Frank Duncan, Karl Fogel.
(Jason Shapiro was unable to attend today.)
-
#257 (call-center dashboards)
In progress. Dec. 19th should be fine.
-
DCSOps OAuth SSO configuration
ARC is folding the old SSO system into one enterprise SSO system. They'll set up the same OAuth connections for DCSOps, so that they're identically configured on the new system as they are on the old. So in principle this should just be a matter of changing "rco-sso.redcross.org" to "sso.redcross.org"
Agreed that we'll do arcdata-staging.herokuapp.com first (configure it to talk to sso.redcross.org, and configure sso.redcross.org to respond correctly, and make sure that works). Then once that works we'll just do the same dance for production.
So Dale will
-
Enable the new sso.redcross.org SSO server to respond correctly to OAuth requests from
arcdata-staging.herokuapp.com. -
Disable the old rco-sso.redcross.org SSO server so that it doesn't respond usefully to OAuth requests from
arcdata-staging.herokuapp.com. That is, we want to test the failure case too. -
Once the above is working, do the same two steps with production
arcdata.herokuapp.com
We'll do the patchover method.
-
-
Rails 5 upgrade: will do after #257.
Confirmed that this is fine. Implicitly, we'll also do it after the RC Care work.
-
#301 (reporting dashboards)
-
RC Care / DCSOps connection
Meeting is set for 8:30am CT / 9:30am ET tomorrow. API service account is ready, and Peter Cartwright is going to walk us through what we need to know. (Note: this meeting happened as scheduled, and Frank put the notes for it above.)
Regarding procurement, Darlene will set up Coupa to accept invoices for RC Care work. She may still need to get an FCC string for it.
-
Re older invoices, including Smoke Alarm.
Darlene sent Andy an email. This is all happening in DCS Finance.
-
Clara has been all through Zendesk tickets.
All the ones that need OTS attention are now assigned accordingly, so Frank should just be able to log into Zendesk and go straight to the tickets he needs.
-
Check in on #173 (damage assessments)
Answer: It depends on (and is a result of) the RC Care switchover.
-
Planned contributor solicitation in early 2021.
We may do a banner notice in Q1 2021 soliciting technical participation in DCSOps enhancement from volunteers. Some have already contributed before, e.g., Jason Winget, Bernie Nazari. There's probably more potential there, if we solicit for it.
-
Review of all items from the FY21 SOW.
-
System
-
IN PROGRESS: Update Rails to to the latest viable version. (related to #157) (large)
-
DONE: Update third-party upstream open source code dependencies (e.g., Bootstrap, Font Awesome) to the latest viable version, both for behavioral reasons and to strengthen the application's security profile. (medium)
-
PLANNED: Move application code to the
redcross/dcsopsrepository on GitHub. (small) -
PLANNED: Change the transactional e-mail provider from Postmark to SendGrid. (#213) (small) (Note: This depends on the Rails 5 upgrade.)
-
IN PROGRESS: Data access (e.g., on-call information) for third-party call centers. (#257) (medium)
-
IN PROGRESS: Built-in report pages, to replace SiSense Periscope dependency. (#301) (medium-to-large)
-
-
Incidents
-
DONE: Create a "Response Status" field separate from "Incident Types". (Not related to the Heroku upgrade, but should be included in this phase of work because it is causing issues in the system today.) (#205) (medium)
-
DONE: (But may need to be undone!) In Incident Details, make changes to the "Damage Assessment" section to reflect the Red Cross' current Direct Client Assistance (DCA) model. (#173) (medium)
-
DONE: For attachments, increase the file size capacity. (#204) (small)
-
PLANNED: Add to Regional Config and Regional Admin capabilities the ability to change an incident report’s date. (#262) (tiny-to-small)
-
PLANNED: Modify the IIR template (Good issue for a new contributor to try; OTS can help with testing.) (#260) (small-to-medium)
-
PLANNED: When calculating mileage, sum for only ‘Team Lead’ and ‘Responders’ (#259) (tiny)
-
DONE: In the incident Timeline, increase the font size of entry labels and the space between entries to improve readability. (#203) (tiny)
-
PLANNED: In the Dispatch New Incident script, modify the wording of two 'questions' and add a Notification Source pop-up menu. (#263) (tiny-to-small)
-
DONE: Hide `Team Lead Trainee' from the Response pop-up menu. (#261) (tiny) (Note: This is actually already done; it was somewhat urgent, and it was easier to just do it than to keep tracking it.)
-
-
Contact Info
-
DONE: On the "Contact Info" button, add the phrase "Text messaging not enabled" to encourage volunteers to set their mobile phone preferences. (#189) (small)
-
DONE: Move "Contact Settings" up under "Contact Info" to increase visibility. (#190) (tiny)
-
DONE: Change wording of "Contact Settings" message. (#191) (tiny)
-
-
DAT Scheduling
-
DONE: Add a "Shift Notes" link (https://www.dcsops.org/scheduler/shift_notes) to the "DAT Scheduling" page. (#207) (tiny)
-
DAT Scheduling (main page)
- DONE: Update "Welcome" language. (#186) (tiny)
- DONE: Remove Spreadsheet links. (#187) (tiny)
- DONE: Fix malfunctioning "Show Roster" hyperlink. (#148) (medium)
- DONE: Next to "Current On-Call Team", enlarge the "See all" link to the size of the adjacent text and make boldface. (#192) (tiny)
- DONE: On the page accessed by clicking on the "See all" link, delete "On Call Schedule", the days, months, dates and time block. (#193) (tiny)
- DONE: Under "Current On-Call Team", delete the days, months, dates and time blocks. (#194) (tiny)
- DONE: Move "Available Shift Swaps", up under "Your flex availability". (#195) (tiny)
- DONE: Move "Your Recent Responses" up under "Available Shift Swaps". (#196) (tiny)
-
-
Administrative Console
-
TBD: Calculate time spent responding for each responder and export the result to Volunteer Connection. (#215) (No estimate, because this work will probably just be done as part of issue #215, as described in the FY20 SOW.)
-
IN PROGRESS: Per-regional-admin access controls. (#166) (medium-to-large)
-
PLANNED: Fix application crash when attempting to add a New Price List Item. (#264) (small)
-
-
Darlene, Clara, Jason
-
Rails 5 upgrade.
Deploy to staging this week, so Clara et al can test while we're working on issues #257 and #301.
-
Karl to review all FY21 items and report back next week.
This is just to check that we've got everything on the list done or planned. Clara pointed out that issue #173 (currently closed as a "do not do") may come back and need to done after all, as new damage assessment classifications may be coming out as part of the move to RC Cares.
-
Clara will finish reviewing Zendesk support tickets.
-
RC Cares
We have the regional go-live schedules now. We haven't yet heard anything back on our request for an API account in RC Cares, but then again we sent the request right before a long holiday weekend, so let's give it some time.
We will probably not be able to complete the RC Care linkage with DCSOps before region Group B switches over to RC Care next week, so OTS to set up a banner in DCSOps to let people know that CAS links will break. Karl will run the wording by Clara, then put up the banner as soon as she approves it.
-
Procurement check-in.
Darlene is having meetings regarding DCSOps hosting and maintenance invoices from FY20 and before and Smoke Alarm invoices. Note that DCSOps hosting and maintenance invoices from July and after -- that is, for FY21 -- have been and can continue to be posted to Coupa.
Almost done responding to all Zendesk tickets. Clara is back this week. Frank will send an overall status email to Clara.
Frank's plate now, in order of priority:
-
Rails 5 upgrade
We want to stage this first, so that ARC can evaluate it while we work on other things.
-
Issue #257
Third-party call-center dashboards issue.
-
RC Cares (assuming we get API access)
-
Issue #301
Reports.
Rupa Hicks, Peter Cartwright, Darlene Johnson, Frank Duncan, Karl Fogel
Peter works for IT Enterprise Architect at ARC. Pete has worked on RC Care since the start of the project and is very familiar with it. RC Care is Salesforce-based. The customizations were done by a vendor, together with internal COE (Center Of Excellence) at ARC. ARC owns the object model and all the code, and COE has the ability to add anything that is needed.
Some RC Care terminology:
-
"Event ID": a human-chosen human-readable name for the event in RC Care. Uniqueness is enforced by the system. We can test validity via an API round trip: any given string will return either zero or one corresponding Event.
-
"Event Number": the "EV-NNNNNNN" code, e.g., "EV-0000003". This is generated by the system, but it's still human-readable, and it is shown to humans in the UI.
-
"Record ID": The Salesforce internal record number for an Event object. This is a hexadecimal blob and is not really human-readable; it is, however, used in the URL for an Event:
https://rccare.lightning.force.com/r/ARC_Event__c/<<<RECORD_ID>>>/view
Data migration: all open CAS cases that have been modified within 60 days will be ported over to RC Care, but this will be done on a region-by-region basis over the month of December. There will be a mapping from CAS numbers to RC Care identifiers, but we may not have all of that mapping at once, since the porting is done on a rolling basis. The first regions are switching on Monday, Nov 30th (official go-live date is Dec. 1).
Todo items:
-
Karl will send Pete a formal request for an API service account.
Done:
From: Karl Fogel To: Peter Cartwright, Darlene Johnson, Rupa M. Hicks, Frank Duncan Subject: Request for a read-only API service account in RC Care. Date: Tue, 24 Nov 2020 14:03:05 -0600 Message-ID: <[email protected]> -
Pete will get us the service account we need for making API calls.
-
Frank will stage the necessary changes in DCSOps.
Exact rollout date still TBD; there are arguments for early in December, and arguments for late in December. OTS will check back with Rupa and Darlene after we've gotten the API calls working, by which point there will be fewer unknowns.
-
Rupa will send us the first regions launching in RC Care.
This is so we can see when the first overlap with DCSOps-using regions is. (Note: we received this on 30 Nov 2020.)
-
Darlene will get from Rupa the right FCC (i.e., account string).
This is the account string OTS will bill against for this work.
Karl will go through Zendesk tickets and respond where appropriate. Frank will be going through outstanding items on Thanksgving. Clara has been out this week, and that combined with Frank working on some other stuff has led to user requests not being responded to as quickly as we (and they) would like.
Follow-up note from later in the day: Karl has now done the above, leaving a comment/question in every open DCSOps Zendesk ticket:
- #243
- #231
- #216
- #205
- #268
- #267
- #266 (followup to #250)
- #265
- #264
- #263
- #262 (duplicate of #264)
- #261
- #260
- #259
- #258
- #257
- #256
(I also read #173 but didn't comment.)
Rupa Hicks, Darlene Johnson, Frank Duncan, Karl Fogel
Rupa runs ARC's flagship client financial assistance program, which CAS is the information management tool for. They're transitioning to RCCare, which will replace CAS. The replacement process is staggered, region by region: some regions are going live in a week or so, though.
In CAS, an Event has a name (letters and numbers) and an identifying number (that's all numbers). A DCSOps user hands DCSOps the former, and DCSOps screen-scrapes CAS in order to convert it to the latter and then present a URL within DCSOps linking out into CAS (because Event URLs in CAS contain the ID number, not the Event's name).
CAS will stay available for some time, but it will be read-only. Only about 10k CAS cases will be migrated to RCCare; the rest will only be available in CAS. CAS itself will not be offering forward references to RCCare.
We should talk with Peter Cartwright, ARC's Salesforce Enterprise Architect. He is doing the mapping from CAS to RC Care, and can answer all our technical questions. Darlene will try to set up a conversation for as soon as possible between Peter and Frank. (Karl will join if he can, but the scheduling shouldn't block on him.)
Note that Clara and Frank have already been talking about this transition, so there is general awareness that DCSOps will need to be adjusted to refer into RC Care instead of CAS.
This work will not interfere with the Dec. 19th deadline for issue #257, because it can take a back seat to the #257 work if necessary.
This meeting was canceled because a couple of people were going to be out and work would still be in progress anyway. We'll meet on the 30th.
Darlene, Steven, Jason, Clara, Karl
(Today was the last of these meetings for Steve. Good luck, Steve!)
-
Yes, that's happening. Clara said that a meeting had just happened today with Frank and the third-party call center folks, in fact.
-
Rails upgrade #126
On track. Upgrade date moved from Dec. 5th to Dec. 12th, due to Clara being out for a few weeks.
-
Support tickets that need OTS involvement (appx 3?).
Frank is on them.
-
Let's transfer
dcsops.orgdomain to Red Cross.Before we dig into why DNSimple (the current domain registrar) is charging $27/month for "Domain professional services" or whatever, let's consider just transferring
dcsops.orgto ARC.Jason will investigate the desirability of such a transfer and let us know next week what he finds out.
-
Smoke Alarm
Domain transfer is still in progress. Site is already forwarding, because of the server-side redirect put in place by OTS, and that forwarding should continue (perhaps via domain-level forwarding) once the domains have been transferred over to the Red Cross.
Re invoices: Andy and Darlene met with Tish Whitaker (who is taking over Ahmed's responsibilities). There will be an internal meeting soon with Darlene, Jason, Andy, Clara, and Steve about this. The recent departure of Ahmed has slowed things down a bit, of course.
-
Check possible staleness of banner notice on DCSOps.org.
There's currently a banner notice on the
DCSOps.orgsite that starts out " The DCSOps deployment was completed at 6:07AM CDT this morning. ...". Karl suspects that this notice might be out-of-date by now, and is checking with Frank.
Notes not public; see ref:c358f513 in OTS internal records.
Just a quick review of what's up next:
-
Ruby on Rails upgrade.
- Respond in email thread with Clara about timing.
- Set up the staging site so they can check it over.
- Actually do the upgrade on December 5th.
-
Issue #257 (third-party call center dashboards)
(Followed by issue #301, then issue #166.)
Darlene, Heather, Clara, Steven, Karl
(Jason was unable to attend today due to a conflict.)
-
DCSOps domain renewal coming up; OTS will handle.
Get documentation for domain ownership and renewal fee, and send to Darlene along with reiteration of ARC's ownership of the domain.
-
Frank will start on issue #257 this week.
-
Frank will work on Zendesk support tickets with Clara this week.
In particular, Zendesk tickets #251 and #243.
-
Frank will confirm the Ruby on Rails upgrade schedule with Clara.
Karl and Frank discussed this right after the meeting. Frank will follow up with Clara and confirm that the early December date works.
-
GetASMokeAlarm sunsetting status:
-
Redirect
www.getasmokealarm.orgtowww.soundthealarm.orgKarl did this right after the meeting. Note that the latter itself forwards to https://www.redcross.org/sound-the-alarm.html, so that's what will be in the URL bar of the browser in the end.
-
Send all data to ARC.
Karl sent this data right after the meeting.
-
Send domain transfer authorization codes to Darlene and Ivan.
Karl sent the auth-transfer codes for the following domains to Darlene and Ivan right after the meeting:
- getasmokealarm.org
- getasmokealarm.com
- getasmokealarm.net
- getsmokealarm.org
- getsmokealarm.com
- getsmokealarm.net
- getasmokedetector.org
- getasmokedetector.com
- getasmokedetector.net
-
Dig up any contract / purchase order paperwork, and complete all outstanding Smoke Alarm invoices. Send everything to Darlene and Andy.
Karl is still working on this.
-
-
Unanimous agreement that Latish and Steven departing is a Big Deal.
This was established at the beginning of the meeting, possibly before Steven dialed in.
Darlene, Andy, Jason, Jake, Clara, Steve, Latish, Ivan Chou (briefly)
-
GetASmokeAlarm sunsetting.
-
Redirect
www.getasmokealarm.orgtowww.soundthealarm.org -
Send all data to the
HFCDataaddress (to avoid spam harvesters, I'm not including the domain here, but it's the expected one). -
Send domain transfer authorization codes to Darlene and Ivan.
-
Dig up any contract / purchase order paperwork, and complete all outstanding invoices. Send everything to Darlene and Andy.
-
-
DCSOps Ruby-on-Rails upgrade.
Clara and Frank are now talking to schedule this. Frank confirmed that it could be around 20 minutes of downtime, though we'll try to minimize downtime of course. They are discussing just putting DCSOps into read-only mode during the upgrade, so that at least queries continue to work the whole time.
-
DCSOps 3rd-party call center support. (issue #257)
OTS confirmed that this will be done well before December 19th, which is the cutoff date for SiSense / Periscope.
-
DCSOps data dashboards. (issue #301)
This will also be done before December 19th, assuming the queries are not very exotic (we have no reason to think any of them would be). OTS does need to gather details about the dashboard queries ARC wants. Karl will start that conversation by email.
-
DCSOps access controls (authorization) for data dashboards (issue #166)
We confirmed that this is less urgent than the two previous items, and that it can happen after they are done (and after December 19th).
-
DCSOps FY21 invoicing.
Use the same FY20 PO number for FY21 work in Coupa (just assign it to the new FY21 line item).
-
Will email Clara and Steven about upcoming Ruby-on-Rails upgrade.
-
Starting on Issue #257 early this week.
-
Some Zendesk support tickets need Frank's attention.
Darlene, Jason, Clara, Steven, Karl
-
FY21 SOW approved; time to start the remaining work.
Karl and Frank will plan the Ruby-on-Rails upgrade in DCSOps, and then Frank will coordinate it with Steven and Clara so as to ensure that any downtime is minimized and happens at an acceptable time. We'll also plan the three Periscope-related issues (#257, #301, and #166).
Karl will email Steven and Clara after the above conversation with Frank, to get things under way.
(Done the next morning:
From: Karl Fogel To: Steven Meissner, Clara Walker, Frank Duncan Cc: Darlene Johnson Subject: Ready for DCSOps Ruby-on-Rails upgrade planning. Date: Tue, 27 Oct 2020 10:12:26 -0500 Message-ID: <[email protected]>)
Ahmed will set up the invoicing code in Coupa soon, likely today or tomorrow. Multiple invoices against the SOW is okay; OTS doesn't have to invoice all at once.
-
GetASmokeAlarm.org
ARC is having a meeting this week to figure out the details of the sunsetting. OTS's preference would be to just transfer the domain(s) over to ARC, who can then forward to wherever is appropriate. ARC is also discussing the outstanding invoices internally; there's a meeting about that soon.
Darlene, Steve, Jason, Heather, Clara, Frank, Karl
-
FY21 SOW.
Has received official approval from project management office. Budget has been approved; Darlene just needs new accounting string. When she gets it, she'll let Karl, and he'll send an invoice (via Coupa) for work done against FY21 SOW, and will send the FY21 monthly invoices (i.e., July onward) too.
-
GetASmokeAlarm sunsetting and status.
Karl will send the full set of outstanding invoices to Latish, Kevin, Darlene, CC'ing Steven and Clara and Jason. As Smoke Alarm was not an IT project, Karl will make sure to include language in the email saying he knows that. The purpose of this mail is to give Darlene, Steve, et al the ability to make the appropriate inquiries internally (i.e., it's not to invoice their department).
OTS commits to supplying all Smoke Alarm request data to ARC in CSV form on request, regardless of invoice payment status.
Darlene and Steve will work with Kevin Kelly to get clarity on exactly when & how the sunsetting is happening.
Note: Karl sent the invoice status mail the next day:
From: Karl Fogel To: Darlene Johnson, Latish Krishnaveni Cc: Steven Meissner, Clara Walker, Jason Shapiro Subject: All outstanding GetASmokeAlarm.org invoices. Date: Tue, 20 Oct 2020 17:25:03 -0500 Message-ID: <[email protected]> -
Functionality to replace DCSOps usage of SiSense / Periscope.
Priority order is:
- #257 (third-party call center data access)
- #301 (built-in reports to replace external query dashboards)
- #166 (data dashboard access control for admins)
Once OTS has green light (invoicing go-ahead from Darlene) we'll start on #257 first, as it's the highest priority. That would need to be finished by the end of the year. Most of the work is really about negotiating with the third-party call centers and Volunteer Connection, making sure that everyone agrees on how the access is going to work. The actual technical changes are minor, says Frank.
-
Reunification system proposal status.
ARC is still evaluating all the proposals. Karl talked to Frank about system design, right after this call, and then sent Latish an email with more details about what OTS proposes to build:
From: Karl Fogel To: Latish Krishnaveni Cc: Steven Meissner, Nicole Van Gilder Subject: Re: [EXTERNAL] Re: LOE for Safe and Well application Date: Mon, 19 Oct 2020 17:52:02 -0500 Message-ID: <[email protected]> -
DCSOps status and support.
We didn't actually discuss this in the meeting, but just to give a status update here in the notes: things seem to be going well. Clara and Steve have most issues well in hand; Frank comes in to help with certain things when they request it. We should probably check in on this in the next meeting, just to make sure.
-
Moving back to weekly meeting cadence.
We agreed to go back to weekly meetings, now that we have things to check in on every week. Darlene proposed
Karl will start a discussion with NHQ, and Frank will hold off on any exceptional SiSense-related efforts until that discussion happens. For more information, see this thread:
From: Karl Fogel
To: Darlene Johnson, Ahmed O. El-Aref, Latish Krishnaveni
Clara Walker, Steven Meissner, Heather Johnson, Jason Shapiro,
Subject: DCSOps status meeting, and direction on SiSense / Periscope.
Date: Fri, 09 Oct 2020 16:26:05 -0500
Message-ID: <[email protected]>
Meeting with Clara and Steve on Friday went well. Frank is going to take on some of the Zendesk items where it looks like the ticket should be escalated to him.
Frank will create an OTS Role account in Zendesk.
Frank will rename arcadata to DCSOps.
Frank will do the "tiny, tiny" #215.
Darlene, Ahmed, Andy, Heather, Steve, Jason, Frank, Karl
-
OTS support for zendesk tickets 198, 173, 205, and 207 coming.
Frank will handle them in the next 24-48 hours.
-
PMO site says FY21 project is still in Finance.
Darlene is asking them for updates; will let us know as soon as she has news.
-
Moving meeting to monthly cadence, at 4:30pm ET.
Darlene has updated the calendar to this new schedule.
We'll arrange topic-specific meetings as needed. For example, once FY21 project is approved, we'll have a meeting to discuss the proposed work that would require change orders / amendments (e.g., dashboards and/or CSV downloads to replace SiSense/Periscope).
Darlene, Ahmed, Andy, Heather, Clara, Steve, Jason, Karl
-
Clara and Steven are ramping up on DCSOps support.
For the next month or so they'll be familiarizing themselves with all the DCSOps support tasks that ARC normally handles (i.e., the things that had been handled by Jim). They'll ask OTS whenever they have questions. OTS will track how much extra work that involves. We expect that it will go down and that pretty soon OTS will be back to just handling the things that actually need coding or back-end technical knowledge, and Clara and Steve will be handling the rest.
Steve mentioned that there is a possibility that he and Clara might be so busy with urgent demands on their time that they have to occasionally offload some of the DCSOps support work to OTS, not because they don't have the knowledge to handle the requests but because they simply don't have the time. We discussed the implications of this, and settled for now on assuming that it won't be like that, and that soon ARC will be able to handle all the Tier 1 and Tier 2 support requests. If that turns out not to be possible, then we'll revisit, since that kind of day-to-day user support doesn't really fall into the maintenance and operations SLA that OTS is currently responsible for.
Darlene will send Clara the template for the IT service desk (i.e., "Tier 0") to use.
-
DCSOps access for people at NHQ and in non-DCSOps-using regions.
(See ref:9548aaea from 2020-08-24 meeting.)
This is done: Jim gave the
dcsopsaccount admin privileges. -
Procurement stuff
-
FY21 project status.
PMO site says the project is waiting in the Finance queue; there were a few questions, which Darlene is answering.
-
SiSense/Periscope replacement project
We'll look at this again after FY21 is approved. It would be done as a change order to the FY21 project.
-
Regular maintenance and operations SOW
Tied to FY21, so wait for that to get approved before sending any invoices.
-
-
Rails upgrade done, but we're holding off on deployment.
The upgrade (from 4 to 5) is fairly major, and involved upgrading a lot of dependencies. Thus there could be some glitches when we deploy it, and since DCSOps users already just went through the experience of all the realignment-related glitches, we want to give them some stable time before anything else happens. Getting the Rails upgrade deployed needs to happen eventually, but it's not immediately urgent -- it doesn't affect features, except maybe a couple in minor ways.
-
Keep regular meetings through the completion of technical work.
Basically, through the deployment of the now-code-complete Rails upgrade and a few other minor things. After that, we may switch to a less frequent cadence, once we're in more of a maintenance mode.
-
Handling regular support requests
Regional Admins (who are listed in Zendesk) need to be pick up most of what Jim did; frank should not pick up all of that stuff. We don't want the SPOF to be a non-Red-Cross employee. Frank does have visibility into all Zendesk tickets, and gets notifications. Right now he just lets Regional Admins handle them.
Frank will switch Zendesk account to the OTS "dcsops-support" role account, which Frank is now a member of.
Karl will follow up with Steve Meissner.
-
Rails upgrade status.
Code is upgraded to Rails 5, but it's not in Production yet.
Frank: "This change is more massive than all of the other changes combined."
All tests pass now (except some known intermittent failures that have nothing to do with these changes).
We will delay upgrade of Production until current post-transition chaos has settled down a bit -- people need a break. Also, Steve and Karl have to establish what the support routes are going to be.
We need someone to look at the upgraded version in Staging and make sure stuff works the way it should in production.
-
Still waiting on Periscope replacement proposal
-
Moving to sendgrid (#213) is blocked on the Rails upgrade.
-
Recalculating time spent responding (#215)
Can be fixed on main and pushed to production.
-
Renaming repository to
dcsops.Yes, let's do it now.
Steven, Darlene, Andy, Jason, Jim, Karl
-
Lots of production support since the switchover.
As people run into issues, Jim and Frank solve them. Frank was tracking Zendesk with Jim until very recently; Frank has four of those tickets left. When an issue is something other than a production instance configuration problem, then it gets a GitHub ticket -- but that's really been pretty rare.
There have been about 120 Zendesk tickets total since the switchover. There are a lot of notification issues, in part due to VC adjustments that need to be done, which requires coordination between regions and DWE (Disaster Workforce Engagement -- "volunteer HR for disasters", as Jim put it).
(Also, Frank and his family moved this week while all this was going on. Whoa.)
-
Jim will get Steven access to the Zendesk tickets.
-
SiSense Periscope
The query is now updated, so the Periscope dashboard pages are able to query DCSOps to display shift information. But it turns out this is still not good enough, because Periscope doesn't auto-refresh frequently enough: it can be a few hours out of date, which means someone at a call center has obsolete shift information and ends up calling the wrong person. (We didn't know about this aspect of Periscope until we made these changes.)
Jim and Frank will implement the following workaround:
The IDMT Dashboard shows a shift window that's longer than the Periscope refresh period, and that works well for them. So for now we can do the same for other regions. This is still not ideal, because it means that now someone at a call center can't just go to Periscope and see "who's on call right now". Instead, the page will show several shifts, of which the currently active shift will just be one -- so the person will have to do a little more parsing to figure out who to call. But this is okay as a workaround.
-
Issue #298 is done.
"When filtering for incidents by County on the More Incidents... page, pop-up menu doesn't display counties"
-
Discussion about extra-regional users having access to DCSOps.
Could Josh at NHQ have a "secondary region" in VC in order to get access to DCSOps (by making his secondary region be one of the DCSOps-using regions)? Well that won't work because DCSOps only supports users whose VC "primary region" is a DCSOps-using region.
But it might be nice if people from other regions and/or from NHQ could get some kind of access to DCSOps -- e.g., it would be useful for Steven. At this point Jim had already had to hop off the call, so we will discuss this at next week's meeting when Jim's there. (Reference marker for use by that next meeting: [ref:9548aaea].)
-
Discussion of updated project
DECISION: Let's just get the current (original) project all the way through first, and look at doing a change order afterwards. For reference, the proposed updated SOW is in this email:
From: Karl Fogel To: Ahmed El-Aref, Darlene Johnson, Steven Meissner, Andy Evans, Heather Johnson, Jason Shapiro, Jim McGowan Subject: Proposed updates to FY21 SOW for DCSOps. Date: Thu, 20 Aug 2020 17:59:30 -0500 Message-ID: <[email protected]> -
Next week's meeting will be Friday, 4 September at 3:30pm ET.
We moved it because Karl will be out on vacation until about Sep. 3rd. (Andy will also be out, but he won't be back by the 4th anyway.)
Most of the notes from this call are included in the notes for the next call. Also: Rails upgrade will follow the support period.
Andy, Steven, Heather, Ahmed, Jim, Karl
-
The switchover happened on Saturday. Realigned DCSOps is now live.
Jim and Frank worked together by video from 3:30am until 2:30pm. (Sound of jaws dropping around the room.)
Here are a couple of technical details that I didn't go into in the meeting but that I wanted to have a record of:
In Heroku, new apps get provisioned with PostgreSQL DB version 12 by default nowadays, whereas DCSOps had been on PostgreSQL 9. Thus at deployment time for realigned DCSOps, we had to choose between upgrading the application to use Postgres 12, or downgrading the Heroku environment to Postgres 9. Naturally, we chose to upgrade the application to 12, in the interests of being future-oriented, especially with the completion of the Rails upgrade coming up.
Another thing: we're leaving archived old data in place (for searchability) and there was some disentanglement of old vs new data necessary during the upgrade process.
Finally, there were various VC complications, but that's all getting sorted out. Jim is handling the bulk of this work.
-
What's pending after the switchover.
-
82 original tickets are now down to 20 or 21.
Mostly it's things like a person expecting to see a shift that they can't see, etc. Often a VC-side adjustment is the solution.
There's also a lot of stuff coming in that's not really realignment-related.
Jim says: "The biggest effort we have now is rebuilding all of the notifications for all of the regions." E.g., An Incident Report gets created, and N people should get an email or a text. But because there are new geographies, new positions, and new shifts, getting all notifications updated will take the rest of the week.
-
Some VC adjustments needed for Northern California
Volunteers in 3 Northern California counties should move from Gold Country to NCCR. This is actually a Volunteer Connection issue; we just identified it for them because it (of course) became relevant to DCSOps during the realignment.
-
Periscope query configuration
OTS (Frank) is updating all the Periscope queries now.
-
Issue #298
This is about updating how a dropdown menu gets populated.
-
Someone's showing up on a shift that he shouldn't be on.
Person was on a shift in the old DCSOps. That shift doesn't exist in the new DCSOps, but the person is still showing up as being on that shift. As Jim put it, it's a "back to the future". We're still tracking down the root cause.
-
Internal OTS task: put updated datasheets under internal version control
-
Rails upgrade is still coming.
4.2 -> 5. This is urgent to do before Jim leaves, so thorough testing happens. (The 5->6 upgade will be a smaller deal.)
-
-
Need for continued in-house DCSOps expertise and direction.
After September 4th, Frank Duncan at OTS will be the only available expert on DCSOps administration and configuration.
Ideally there should be someone at ARC who knows DCSOps well and makes decisions about it. Most likely that person would also be the in-house expert whom regional admins go to to solve problems.
This is ultimately an ARC internal decision. We just discussed the topic today to get it on everyone's radar screen.
One good thing is that regional admins will be a bit more mutually connected than they have been. They now have a forum, which Jim just created, and they are aware that they'll have to take more ownership of DCSOps. But still, they're a group, and can't provide the unified direction that comes from having a dedicated in-house product manager.
(One other thing that would help: implementing the feature described in issue #166, ("Per-regional-admin access controls").)
-
SiSense / Periscope / call centers
Periscope costs a lot per month, so if we move all the call center reports and other reports into DCSOps, the cost of doing so will be paid back very quickly.
-
Update FY21 SOW
Karl will see what's left in the budget, update the SOW, and send both the updated version and a list of changes against the previous version. Recipient list: Ahmed, Steven, Andy, Heather, Darlene, Jason Shapiro, Jim.
Everything we discussed in this call is now reflected in the notes for the regular NHQ check-in meeting that happened later in the day.
Jim, Karl
DECISION: Go with Periscope short-term solution for now.
Jim will create a Periscope account redcross-periscope (i.e., the
expected role account address at OTS) for us, and give us whatever
permissions we need.
Note that periscope may not be as trivial as we think. The queries are complex; getting time zones right can be tricky. Ideally, DCSOps should be on UTC internally, and then applications display times based on user-specific or client-specific configuration, but instead (we think) each DCSOps region has its own time zone or something.
We discussed just sending a similar email to the one we send to DirectLine (3rd-party call center for Cascades, Gold Country, and NCCR). That email contains has two files: a roster (list of names), and a schedule (mapping person IDs to shifts). DirectLine parses that email. But the problem is, there's no way these other call centers would be able to implement parsing a similar email before Saturday. They just need a page they can log in to; hence the Periscope solution.
In the long run, if we give DCSOps the ability to do everything that "currently" (i.e., as of Switchover Saturday) is done with Periscope -- that is, including not just the three proposed call center Periscope pages but also any other reports we extract via SQL query from DCSOps and display in Periscope -- that will save the American Red Cross money, because Periscope costs a lot per month. Karl will raise this topic at the next NHQ group meeting.
Jim, Frank, Karl
Third-party call centers need dashboards they can look at to know who's on call, i.e., who to call when they get an incident.
Currently, this is the situation for three regions that are using call centers:
- MN-DK: Using Jason's custom page (generated by scraping DCSOps; see issue #257)
- ID-MT: Using a Periscope query.
- KS: Logging in to DCSOps (which means they get too much access)
(Note that there's also DirectLine, used by Cascades, Gold Country, and NCCR regions. That case is a little different: DCSOps sends DirectLine an email which DirectLine parses. This isn't changing, so it didn't come up in this meeting.)
Short-term: have everyone use Periscope, and reconfigure the Periscope queries as needed for the upcoming schema changes.
Longer-term: it's a "Medium"-sized task to enhance DCSOps to have the access-restricted pages and corresponding administrative pages such that each call center can log in to DCSOps and see exactly the information it needs to see.
Periscope usage in general is going to break, by the way. Jim knows about this. We can't fix that all before the 15th. However, we can help update SQL queries after the switchover.
Andy, Jason, Ahmed, Steve, Darlene, Jim, Karl
-
This Saturday, 15 August, 4am Central, is go-live date for realigned DCSOps.
As of this afternoon (10 August), every region will have had the opportunity to express their concerns. If something's not working for a region, it will get fixed after Saturday. (Note that Nebraska has been unresponsive; if anything turns out to be broken for them, then we'll have to fix it after the fact.)
We fully expect some post-deployment fixups to be needed. Jim expects that 90% - 95% of the issues people have wil to resolveable by Jim and his team, without involving any actual programming.
-
Front page of DCSOps will have three new buttons:
-
Help videos, covering admin tasks people might not have done recently but that may be needed after the realignment.
-
Link to a page that says where to get direct technical help, and provides a few FAQs at the top.
-
What's New in DCSOps" listing new features and important notices.
-
-
Jim provided the up-to-date email address for support.
-
Jim has emailed the VC team re login changes.
Said please don't make any changes right now. No response, but Jim trusts Diana Chu to know what's up and not do anything disruptive.
-
VC import query processing stack dependency complication.
It turns out this is not a problem after all, whew.
From: Frank Duncan Subject: VC Import in staging To: Jim McGowan, Karl Fogel Date: Tue, 11 Aug 2020 11:50:21 -0500 Message-ID: <CAK=DgqdrPC24X+aqtfo08S=dbQnFov647NSgxy7jR7cecp4UNg@mail.gmail.com> It was not what I thought it was, and so it works :) Planning on upgrading stack on saturday. Frank -
Ahmed is going to the grocery store later today.
Actually, he's says he's cheating and using Instacart.
-
Realignment v1 has been deployed to staging.
We're now getting feedback from regional admins.
-
FY21 changes deployed to staging on 30 Jul 2020.
This means that in addition to the realignment changes, the staging server also has all the changes already listed in the July 28th meeting plus all the changes below:
-
#205: Create a Response Status field separate from Incident Types
-
#189: On the the Contact Info button, add the phrase "Text messaging not enabled" to encourage volunteers to set their mobile phone preferences.
-
#190: Move Contact Settings up under Contact Info to increase visibility
-
#191: Change wording of Contact Settings message
-
#207: Add a Shift Notes link (https://www.dcsops.org/scheduler/shift_notes) to the DAT Scheduling page
-
#186: On the DAT Scheduling page, change 'Welcome' language
-
#187: Remove Spreadsheet links from DAT Scheduling page
-
#148: "Show Roster" hyperlink is malfunctioning
-
#192: On the DAT Scheduling page, next to Current On-Call Team, enlarge the "See all" link to the size of the adjacent text and make boldface
-
#193: On the page accessed by clicking on the "See all" link on the DAT Scheduling page, delete "On Call Schedule," the days, months, dates and time block
-
#194: On the DAT Scheduling page, under to Current On-Call Team, delete the days, months, dates and time blocks
-
#195: On the DAT Scheduling page, move Available Shift Swaps, up under Your flex availability schedule
-
#204: In the Attachments section, increase the file size capacity
-
#203: In the Timeline of incident reports, increase font size of entry labels and the space between entries to improve readability
-
#254: On admin MOTD, use "All Regions" (instead of blank) for global message.
-
-
For an overview of what's on deck right now, see the July 31st internal meeting.
-
Mismatch between current Volunteer Connection data and DCSOps expectations.
Many of the VC positions that we wanted to see are either not yet created in VC, or don't match what we expected them to be (what we were told the names would be), or don't have volunteers/staff actually populating the positions.
This is making testing difficult. For example, we may have a certain capability tied to a certain VC position. If no one occupies that position, then there's no way for us to test that capability in DCSOps.
So the big lift this week is to make sure that VC positions and DCSOps positions are lining up the way they need to. Jim is doing some manual assignments in order to test certain things, although that's a tedious process.
Biggest concern is: are the regions ready in VC, so that when we push these DCSOps changes to production everything works?
-
We're using regular GitHub issue-based workflow to report and handle problems as they are found (on staging).
-
The new VC import query is working fine. Thank you, VC team.
-
VC import is apparently going to change their login method.
We just heard about this, and don't know the details, so we need to find out what they meant by that. It was apparently a throwaway line in one of the emails they sent, something like "And let us know when you're in production so we can disable the old login method." We hadn't been aware of any login method changes.
Jim has asked them not to make any changes without talking to us first, and is still awaiting a response to that.
-
DCSOps VC import query processing stack has a dependency complication.
Upgrading the Heroku machine stack will cause problems for our side of the VC import. This is fixable; it just raises the importance of going from Rails 4.2 to 5 in production before the November deadline.
Details:
Ruby uses a package called
xls2csv, which in turn depends on an underlying C library. The C library's.soversion number is explictly specified in the Ruby gem, and the C library is packaged in the machine environment. This means that upgrading the machine environment is now linked to upgrading Ruby all the way to Ruby 5 -- i.e., we think we can't just stay in our current 4.2 upgrade while moving to the new Heroku machine environment.While we've been able to hack this in our dev environment for the purposes of doing the VC import query right now (i.e., pull staging data down to local dev environment, do the query locally, and then re-upload the now-updated DB to staging), hacking it in a Heroku environment is a rather different matter and probably too complicated to be a reasonable solution.
-
We anticipate that there will be post-production cleanup.
We're seeing enough problems that we know that when we push to production our work will not be over. It's be urgent enough to get a working DCSOps into production that we're going to do that even though we know we'll still be resolving various issues afterwards.
-
FY21 SOW status
On its way through approvals; Darlene is checking on it frequently.
-
Status of current hosting invoices
Karl will send an Apr-June 2020 invoice for monthly hosting and maintenance, and, separately, a July 2020 invoice. The reason the latter should be its own invoice is it is in FY21.
(DONE: OTS internal reference ref:0372a859.)
-
Contribution intake as part of regular maintenance.
We discussed greater code participation from staff and volunteers, especially after the realignment is complete. OTS will continue to view contribution support as being part of our maintenance contract; if the contribution level starts to get high enough that that's ever a problem, we'll bring it up then (this would be in the "nice problems to have" department). Steve, for example, would like to try a couple of "low-hanging fruit" fixes if he can get the time. We also discussed the potential for Code4Good and Code for America participation.
See the third through last items in the next meeting (above). Those are what Frank and I talked about in our internal check-in today.
On deck for Frank right now:
Much stuff, all ultimately related to #255:
-
Broken unit tests need to get fixed.
-
Admin interfaces for the new positions.
-
Preventing historical data from showing up in the UI, which means adding "disablement" functionality and updating UI to respect that flag in the appropriate places.
-
The Google Maps issue (#253).
-
Any problems revealed in testing over the next week, of course.
Then after that comes the Rails 5 work and any remaining FY21 work.
See conversation here.
DECISION: Data (e.g., spreadsheets) that identify particular Red Cross staff or volunteers by name and/or by internal ID number will not live in the public repository. They will be checked in to a private area in SVN, and referred to from the open source scripts that make use of that data.
-
Rails status
4.2 is in staging, on a new Heroku stack. (Hwoever, printing may or may not be working -- right now it can only be tested on production, because the PDF rendering stack is so complicated.)
Todd left us with a clear path to 5. Frank needs to integrate that work with the work Frank has already done, which will be "a task". Todd says that 5->6 after that shouldn't be too hard, it's just that he didn't get there. So plan is after realignment is done, we're going to merge in all FY21 stuff, then immediately after that Frank will do the Rails 5 completion and push it to production. (Jim's vacation starts to be a factor in the timing here.)
-
Schedule
Frank guesses last week in August for actual completion of realignment work. But note that we don't want to then push a Rails 5 upgrade right when Jim goes on vacation -- we want a couple of weeks with Jim around to test it.
-
Dockerization and sample data
Turns out that no one's no one's ever actually tried it. But it would only take Frank a little bit of time.
Karl will tell Steve et al that we can dockerize if that will help.
For non-RC contributors, there would still be an issue about production data. The seed data is not sufficient to test the app (e.g., "Does this look good for New York?" etc). Giving RC people production data is fine, but this won't be an option for some volunteers, or maybe those volunteers would need to sign NDAs.
Darlene, Andy, Steven, Latish, Heather, Jason, Jim, Karl
(Heather Johnson and Jason Shapiro joined for the first time today; they're replacing Rod Tolbert, who's changed positions.)
-
Testing status
Jim met with all the regions yesterday and let them know that they can go in and start testing.
A number of regions' Volunteer Connection positions are not yet ready to synchronize with DCSOps (i.e., DCSOps is ahead of VC in terms of position creation), so there's still some back-and-forth with the Volunteer Services team, which decided not to make any of those changes available until they're all ready, which will happen on this coming Friday.
(However, the queries that we are pulling from VC are working fine, which is good to know.)
This situation leaves are go-live date in flux, given that this means we can't really start testing realigned DCSOps until Friday. Our plan is to produce a list of positions that aren't matching and hand it back to Volunteer Services so they can make any corrections necessary.
Andy and Jim discussed adjustment to go-live date. Currently it says August 1st. Now tentatively pushed out to the 8th.
-
Changes staged for testing on 21 July.
These changes have now been tested and everything seems fine.
-
#253: Move scope map configuration from fusion tables to inline polygon
-
#143, #208: On the main Incidents page, for Recent Incidents, display the street name between the city and incident number
-
#198: After clicking Submit Incident Report on the main Incidents page, bypass the Currently Open Incidents page and go directly to the Create Incident page
-
#209: Change the NorCen Dispatch Console homepage link so that users bypass the NorCen Dispatch DAT Incidents and Disaster Dispatch/New Incident page and land on the Dispatch New Incident page where they can immediately start creating a new incident
-
#223: Paginate the "Currently Open Incidents" page.
-
#199: Create a link on the main Incidents page to access the Currently Open Incidents page
-
#224: Fix the "Current Activity" link.
-
#225: Distinguish between manual and scripted for "Submit Incident Report".
-
#201: Create a way for closing open incident reports listed on the Currently Open Incidents page without having to complete all mandatory fields
-
#140: Permit resending of recruitment messages from the Responder Console and individual messaging to unassigned responders
-
#141: In the Responder Console, increase the size of the Recruitment message window so that the full message can be seen and edited
-
#211: On the Disaster Dispatch page accessed via clicking the NorCen Dispatch Console homepage button and then the Dispatch Console button, create a way to delete the incidents listed under the Dispatching Incidents heading
-
#212: On the page presented after a new incident is created using the NorCen Dispatch Console, modify text
-
-
Potential followup issues
Karl has done effort estimates (and attached corresponding labels) for all of these potential followup issues. They were not part of the FY21 list, but some of them may be small enough that we can just do them, and in any case we're very happy to review any patches coming from others (just as part of regular maintenance). Karl will reiterate that in a followup email after this meeting.
-
#166: Per-regional-admin access controls.
Already estimated -- see previous notes.
-
#257: Provide way for call centers to get list of on-call people.
Comparable in effort to #166 above, but probably somewhat smaller.
-
#264: When attempting to add a New Price List Item, the application crashes
-
#263: In the Dispatch New Incident script, modify the wording of two 'questions' and add a Notification Source pop-up menu
-
#262: Add to Regional Config and Regional Admin capabilities the ability to change an incident report’s date
-
#261: Hide ‘Team Lead Trainee’ from the Response pop-up menu
This would be a good issue for a new contributor to try.
-
#260: Modify the IIR template
This would also be a good issue for a new contributor to try. Steve mentioned that he might have a go at it; OTS will keep an eye out for activity from Steve in the issue or for a pull request from Steve.
-
#259: When calculating mileage, sum for only ‘Team Lead’ and ‘Responders’
-
#258: Create a DCSOps Sandbox for volunteer training
-
-
We expanded the Expedited WAF protection based on more bot attacks.
The bot queried some new paths; we blocked those paths.
-
RC Cares synchronization
Can DCSOps with RC Cares in the same way we currently do with CAS? RC Cares is built on Salesforce, so it certainly could have APIs that are available to DCSOps, but we don't yet know if it will.
Latish will investigate getting a user story about this added to the RC Cares project, so they at least know that this issue exists.
(Note that historical case data in CAS will just be archived, in some way that one can reach it, but the existing "View in CAS" links will break.)
On the agenda was:
-
Changes to One Drive files
Jim will be making changes to the One Drive files, with the changes in red so that he can track, though that won't affect the script code
-
Shift Categories
Shift Categories are referenced in the sheets that don't exist in the system, which needs to be fixed. Jim to take care of.
-
How to record "Monday, 6AM - Friday, 6PM"
This is unclear how to represent in the system, however the scripts can be built with default values, which will be changed after the system is live.
-
Dispatch Configs
These can be scripted, though Jim will update the sheets to add names at a later date. For now, the names will be the shift territories.
-
Outside logins (SSO, Call Centers)
The big question is about information that outside call centers use to get access to on call personnel. Currently there are public pages through periscope and a page that Jason Winget creates, which is not a good way. Jim and Karl to follow up later. Various solutions disccused between Frank and Jim include:
-
Create volunteer connection accounts for these users, with positions that limit them to specifically these pages. VC may not be on board with that.
-
Create a hidden public link - bad security, easy to implement
-
Enable local logins in DCSOps for outside sources that aren't strictly volunteers. Currently impossible to do in DCSOps through the active admin interface, and may be brittle when keeping in step with how DCSOps handles VC imports.
-
Issues 209/212/225 - staging not configured, Frank to fix (done)
-
Attending: Ahmed, Andy, Jim, Steve, Karl. (Darlene out today.)
-
VC synchronization is still disconnected. Everything still works.
-
Internal realignment is now happening.
The regional spreadsheets are done; we're doing the core realignment work, based on those spreadsheets.
-
DCSOps will need a testing period for the realignment.
If Volunteer Connection isn't ready for DCSOps to rejoin it by Monday or Tuesday of next week, our August 1st go-live date for DCSOps may need to be pushed later, because there needs to be enough time for the regional admins to test the staging version of re-aligned DCSOps that we're going to roll out first.
-
Bot attacks are solved now.
We talked to Heroku and got the "Expedited WAF" filters to protect
www.dcsops.orgas well asdcsops.org(we had thought that would happen automatically, but it turns out not). Since then, the request timeouts are no longer happening. So we consider this problem solved now, at least until we see some new incident. -
Issue #166 estimate.
This is about per-regional-admin access controls. We discussed cost (I usually don't put work estimates into the public meeting notes, but it was a fairly small cost, and Darlene received it last week and incorporated it into budget planning).
-
Regions that have external call centers (other than DirectLine in CA) have been creating user accounts in VC so that the call centers can log in and see the schedules. We would prefer to put up a single web page with the data that the call center needs, rather than having them log in with these bot accounts.
Solution: Use a local login (not VC-based login) for each call center, and have a page that call center goes to to get the data they need.
Karl to estimate this for next week's meeting or sooner. He's pretty sure it will be small (e.g., comparable to issue #166 above).
-
Technical support during immediate post-realignment period.
Currently Jim does tech support via Zendesk. For the next month or two, he'd like to create a Zendesk user account for each region. So that's 5x9x2 == $90, for a month or two at the most. That way Jim can assign requests out to the appropriate person during the heavy-support period that's coming up right after the realignment.
Steve and Andy will discuss (with Jim) whether getting in-house support desk accounts is better, but everyone seemed to agree that a solution would be found one way or the other, and just using Zendesk is one plausible solution.
Attending: Darlene, Andy, Steven, Jim, Karl
-
Volunteer Connection import has been temporarily turned off (as planned).
-
Spreadsheets just got completed today; final realignment work can start.
Jim has a call with Frank coming up. The last bits of the (rather complex) New York spreadsheet just came in; Jim will update them, and then OTS starts adjusting DCSOps. We expect there will still be cleanups and tweaks even after DCSOps is realigned, though.
-
Dealing with bot attacks that route straight to IP address.
Some of the bot requests for random paths appear to be coming in via direct IP address connections rather than through a DNS query. If the bot had used a DNS query, it would have been routed through the Expedited WAF filter, and then Heroku would have handled it and DCSOps would never have even seen the connection (and thus the request wouldn't have caused DCSOps any problems).
However, what we're seeing in the logs is requests being made specifically for paths that we know we have filters for in WAF, which means that the requests must not be using DNS queries -- instead, the bots are using DCSOps's IP address directly.
Frank is talking to Heroku's Expedited WAF team about this. They've been pretty responsive in the past, and we expect to have an answer soon. Karl will update us next week (or sooner) about progress.
-
Jim is also working on two other issues.
One that Karl didn't catch (cough cough) and the other about improving map boundaries on main Incidents page.
-
Issue #166 may go on a future SOW.
This is about making DCSOps's access controls more sophisticated, by giving each regional administrator the ability to perform admin tasks for just that region. Implementing this is an important step to making DCSOps group-administratable (which would take some of the burden off Jim, or at least give Jim the option of delegating certain things when there is an administrator available to take them over).
By next week, Karl will have an estimate of how much effort that task will take. If it's a little, we're likely to just unofficially include it in the FY20+FY21 work. If it's a lot, then it will need to be in a new SOW. We don't know yet; today was just laying out the possibilities, and we'll make the decision later.
-
Production service timeouts due to bot path-probes.
In our logs, we're seeing paths being probed that we know we've set up Expedited WAF filters to block. This means that the requesters (the bots) are not using DNS to look up DCSOps (because Expedited WAF interposes its own server via DNS record), but instead the bots already know DCSOps's actual IP address and are just connecting to that directly.
Frank is talking to the WAF team. They've been very responsive; we expect to get a solution from them.
-
Everything in FY20 and FY20 is done except for these:
- Realignment itself (starting now)
- VC Import query update
- Full Rails upgrade
- Postmark->Sendgrid transition (should wait for Rails 6)
-
Jim has sent some realignment spreadsheets.
They weren't done as of this meeting, but note that they were finished later today.
-
Today is the day to turn off the VC import.
-
VC finished writing their new query on Friday.
Frank will start looking at and testing it today.
-
This should get done, but it's not clear it's part of this SOW. Karl will mention this to the folks at NHQ.
Jim just did a dyno restart after DCSOps was showing slowness and showing lots of critical errors in the Heroku Dashboard.
We discussed why Expedited WAF does not apear to be protecting us from those bot probes. Karl speculated that it's because the paths being probed are different every time. (But in a meeting the next day, Karl learned from Frank that the issue is something totally different, though; see those notes for details.)
(Attending: Darlene, Andy, Steven, Jim, Karl.)
-
Jim confirmed that the new scopes and regions have been created.
-
Getting feedback for the staging site soon would be good.
Karl added this to the Detailed Schedule for Home Stretch in the Work Plan.
-
Final realignment spreadsheets expected EOD Wednesday.
New York region is complex and there is some delay in getting that data. Every day of delay in getting the full set of completed spreadsheets translates to a day of delay in finishing the realignment.
-
VC admins are setting up the new query (
_DCSOps_orc_1).We said we would prefer a query to an emailed zipfile.
-
Change projected completion date to Aug. 1st.
We discussed the possibility of delay in getting the realignment spreadsheets, and decided to tell people that things would be done by August 1st (better to tell them that and be early than tell them July 25th and be late).
-
Jim will put up banner notice about VC tonight.
A site-wide notice will tell all DCSOps users that DCSOps will disconnect from Volunteer Connection starting July 13th, and will only reconnect once the realignment is completed. Jim confirmed that the service desk knows about this.
Jim will evaluate the pre-realignment changes currently in DCSOPs Staging. This is mainly the Rails upgrade and issue #185, but we need to make sure that things are all still working as Jim expects (e.g., DAT Scheduling, etc). Jim will nick-flag both Karl and Frank in these comments, just to make sure email notifications go out.
Jim will do that this weekend. Once he gives the green light, Frank will push the remaining pre-realignment changes and Jim will test those on a change-by-change basis.
-
Google Maps issue.
Jim will be getting the shapefile very soon, and will just send Frank the coordinates data in a table.
-
DirectLine status.
MN and Dakotas were going to use DL, but they changed their minds. So it turns out nothing is going to change re DCSOps's DL usage.
Background: DirectLine is a call center in Berkeley, CA that takes initial incidents for Cascades (OR), NCC, and CA Gold Country. DL records incident data in their proprietary system and then pushes an email to DCSOps so DCSOps can ingest it. Also, once or twice a day, DCSOps sends DL an email so that DL has up-to-date roster information. If you look under Incidents->Dispatch Logs, you can see those incoming incidents from DL.
-
Realignment spreadsheet completion for all 10 DCSOps-using regions.
Jim has final meetings with all 10 regions Mon-Wed, to finalize everything. By EOD Wednesday, Jim should have all the information for all 10 regions ready to go, and can hand it to Frank.
-
Repurposing+reconfiguring existing regions vs creating new regions.
Frank says it's definitely okay to re-use regions.
-
What about the meaning of old (pre-realignment) incidents?
Frank is going to run a batch job to reassign all existing incidents to the new (post-realignment) region they would have been in had this new region configuration been in effect at the time.
Any incidents not assigend to a current region after the realignment will be placed in a new "No Current Region" region that Jim will create.
Jim will reconfigure/delete existing scopes as needed. ("Scopes" are abstract layer that sits on top of regions, so that you can arbitrarily create new views into DCSOps. E.g., Jim could create a scope called "Pacific Division" and tell it to cover the four regions in the Pacific, so that he'd have a scope for querying purposes.)
Frank and Jim discussed referential integrity issues and came to a resolution that they seemed satisfied with, although Karl didn't capture it so instead he wrote this rather lame sentence.
-
Discussion of admin console changes.
Jim will go through each of the menu items under "Incidents" and tell Frank, for each category, what to preserve and what not on a case-by-base basis (old Shift Groups / Shift Territories / Positions / Response Territories / etc).
"Volunteer Connection Positions" will go under "Roster". It's the way we match a person to a position. It's a regexp match right now; we're going to make it an explicit mapping that any admin can do. This will be much cleaner. This gives us the ability to recognize new VC positions as they are created.
-
Global MOTD.
Jim will need to post some global announcements in DCSOps about the upcoming chanes. We checked that he can do this; he can. Karl filed issue #254 about changing the admin UI so that this feature is easier to find.
-
Role Scopes.
Jim and Frank had a long discussion about these, that Karl didn't fully understand, but Karl is confident that they understood each other perfectly and agreed on what they were going to do.
-
Schedule of events for the home stretch.
See the Detailed Schedule for Home Stretch in the Work Plan.
Note: Due to time constraints, some bugs found during regional testing of DCSOps Staging may only get fixed after realigned DCSOps is fully in production. Jim will inform regional admin contacts of this as needed, and will help us prioritize which problems must be fixed before realigned DCSOps is deployed into production and which problems can be left to be handled after that deployment.
Staging SSO requires a slight modification to DCSOps to connect to it (the authentication dance is slightly different.) Frank has it working, but there's still a problem: Staging SSO DB is different from Production SSO DB, so we can't use a production copy of DCSOps with it (Staging SSO talks to Staging VC instead of to Prod VC -- this is probably an internal Red Cross decision. And note that Staging VC may or may not be awake at any given time.).
Red Cross tells us that all the usernames in the DB are indeed duplicated between Staging VC and Prod VC, but the users' RCO IDs are different between the two. DCSOps is, of course, using the RCO ID as its primary user key.
Possible courses of action:
-
Best option: Production SSO could respond to Staging DCSOps.
-
Second best: Run query-tool to pull down all RCS IDs for the Staging SSO usernames.
That is, they finish their query-tool revamp on Staging VC, so that we can have Staging DCSOps use the new unified query and fetch the Staging SSO RCO IDs.
But the VC import report from the query tool doesn't work in Staging VC right now, because they're actively developing it to translate it into a single global report for us. There are roadblocks to using Staging SSO w/ Staging VC.
As Frank understands it: We log in to on of 12 accounts for each of the regions, and run
_dcsops1report for that region. DCSOps logs in as that region, navigates to the query tool page, and basically clicks the "download excel spreadsheet" button.In Staging VC, though, the
_dcsops1report is not in the first page. Furthermore, Frank doesn't think that all the regions even have accounts in Staging VC. And the query for Greater Chicago & Northern Illinois is the one they're actively working, so that query is broken (b/c they didn't start on a new query). There are a lot of reports in Staging VC (but not in Prod VC) that are_carol.Once they do finish the one-big-unified-query, things will be great.
Frank would have preferred to test Staging DCSOps w/ Prod SSO, and get that all into production by yesterday, before making the changes for the upcoming new query tool. Instead, we could end up testing everything at once, and that'll be two weeks from now -- which is when the realignment is supposed to be happening at all. Not great.
We're going to end up here in the next few weeks anyway, while testing the realignment; it's just that it would nice to have it sooner so that we could test the DCSOps-side changes before dealing with the big multiple-entities realignment changeover.
-
Third best: If Staging SSO had accounts for all the people Jim will use for testing, and we had that list of Staging RCO IDs (since they will differ from Prod SSO RCO IDs).
Okay, so let's go for "Best Option". Can Staging DCSOps talk to Prod SSO?
TL;DR: Yes, now it can (shortly after this meeting). We asked Dale Siar to make the necessary adustments to Prod VC's expectations about redirect URI. He did, coordinating with Frank who made the necessary adjustments on the DCSOps side at the same time, and it all worked.
Summary:
Everything on track for FY20 changes. Some concern about getting Staging SSO working in time for us to do test deployments.
Details:
-
Jim and Frank coordinated with the VC team on data import.
Instead of 3 reports per region, we're going to get one unified report. Frank will be following up with Jim on this.
The new report will avoid pulling unneeded information, such as inactive people, or people who don't need to use DCSOps (e.g., biomed). With help from Diana Chu, they have cut the size of the data transfer by about half.
(Frank will post his meeting notes in this file later; he's already posted them to the DCSOps Development mailing list.)
-
Still waiting for Staging SSO team to get back to us saying their side is updated. They said they would send that notice today.
Jim has pinged Dale Siar. Karl will send an update about status of this by the end of the week.
-
Creating new Regions: Jim still working on the spreadsheets.
Jim will just create the regions and their configurations in DCSOps himself, and then hand OTS the spreadsheet corresponding to the new regions so we can import the data.
-
Create and display new regional boundaries maps.
Jim has someone who will supply numerical (lat/long) data now. Frank knows how to import it.
-
Allow up to 20 flex responders to be listed on the responder console.
This turned out to be an invalid issue. (And was tiny anyway.)
-
Permit resending of recruitment messages from the Responder Console and individual messaging to unassigned responders.
Jim and Frank talked about this and Frank now has a clear path forward. This is probably a small, not even a medium.
-
Easy way to de-list incidents.
On the "Disaster Dispatch" page accessed via clicking the "NorCen Dispatch Console" homepage button and then the "Dispatch Console" button, create a way to de-list or close the incidents listed under the "Dispatching Incidents" heading.
We had originally thought this was large, for complicated reasons, but Frank found a solution that makes it tiny.
-
Detected and resolved some Twilio (text messaging) issues related to the Rails 4.2 upgrade.
-
Frank posted some notes (in the DCSOps development mailing list) about how events are handled.
-
Confirmed that FY20 is fully invoiced.
-
We will start some of the FY21 changes (beyond the Rails upgrade) now, since there are a number of changes that are both small in terms of coding time and large in terms of user benefit.
-
Volunteer Connect will get us a query, hopefully for all regions
This query will be helped by some relaxed requirements, mainly that we don't need outdated positions.
We may also be able to remove some of the queries that has extra address information (In _DCSOps3 report), as that may be duplicated in the main query. VC to check up on that.
Jim to follow up on second_lang, third_lang being used, and to see if qualifications are used.
Query should be up this week for us to use in testing
-
Discussion about events
Went over email here, with additional information on where templates are in the codebase, and how we can modify them.
-
Issue 140 - resetting stuff for the responder console
The conclusion come to is that a reset is not needed. We just need the ability to re-send the recruitment message, or to send individual messages.
-
Issue 211 - culling dispatch console requests
My solution (noted in the issue works, but extra work can be added that Jim will comment on.
-
Issue 128 Determined that this is working as designed, and the issue can be closed (to be done by Frank)
-
For Maps
Just use comma separated coordinates, from the 'boundary polygon filter' field already in scopes.
(Basically the same as the notes for the regular group check-in today, so please see above.)
Top-line item: Frank is blocked on Red Cross regarding three things:
-
SSO
Scheduled for Monday, because RC SSO is changing at the same time. This has the potential to be disruptive. We are in touch with them about it. Karl will respond in the thread today.
-
New imports from VC
Also meeting on Monday, with Diana Chu and Marcus Mason.
-
Realignment spreadsheet
Should receive from Jim today, or maybe Monday if Jim gets delayed.
Next we went down FY20 list in the Work Plan, making sure every item is up-to-date.
Also: Frank mentioned that Jim has been texting Frank and Karl but using Karl's desk phone number (the 312 number), which doesn't get texts, so Karl hasn't seen any of those texts. That's okay, because Frank saw them and handled everything, but still, we should fix this. Karl will tell Jim about this. (Done, right after this meeting.)
-
Need meeting on Monday, June 29th, to go over exports with VC. Goal is to figure out how they are going to work, and what we need to do on dcsops side.
Frank to come to meeting with information about:
- Which columns are currently being used from exports
- Does dcsops check to see if its information is current so as to not do extra work
- Do we do something about inactive volunteers?
- Do we make the account inactive in dcsops if they came as inactive
Also of note, they're SSO may not make logging in impossble if the account becomes inactive.
-
Jim to create Issue on changes to responders console as needed.
-
Volunteer connection test environment is coming with new scripts.
Can only do a small number of regions, but will play around with it when we have full credentials with the correct staging site.
-
SSO for DCSOps staging is coming
-
Change to Issue #173
The livable/not livable classifications are going away in favor of returning to the previous destroyed/major/minor/affected. There is a new "Uninhabitable" that will be added, but not sure if that's just a flag, or is actually a numeric. Jim to follow up and update issue when needed.
-
Dispatch console
The dispatch console needs to be available for all regions, not just the special
norcen_dispatchscope. The buttons will be moved to the top, and language was defined for that that will be outlined in an issue to be created by Frank.
Attending:
- Ahmed El-Aref
- Darlene Johnson
- Jim McGowan
- Karl Fogel
Notes:
-
RC SSO needs to support DCSOps Staging
This is an important item and a blocker for other items.
The RC production SSO server needs client credentials for DCSOps Staging, so the latter can to authenticate users via RC SSO.
This is so Red Cross staff and volunteers can test DCSOps changes on staging before we deploy those changes to production. Currently, Jim and others can't log in to DCSOps staging. Although in theory OTS could creates a special "local login" on DCSOps staging, that would be problematic in several ways. For one thing, then DCSOps staging wouldn't be using the normal login flow, so it would differ a bit from production, which is never a good idea when testing. Also, we'd need to re-establish all those local logins on staging every time we do a mirror-refresh from production data. That would be awkward and likely error-prone. The best solution is to just get SSO working with DCSOps staging, as should have always been the case.
Status: In progress. See email thread "Need Client ID to test DCSOps with SSO production instance". We asked Vineeta John, who referred us to Ivan Chou, who referred us to Dale Siar and Shana Jerman. (Patty Jordan, Jason Moore, and other people are also on the thread.)
-
Data import process into DCSOps from Volunteer Connection
Status: In progress. We just got credentials to the VC test environment, which we can use to test DCSOps running queries (the new queries that we'll need) to pull data from VC and get that data into DCSOps.
-
Realignment data spreadsheet(s)
Jim says: collecting configuration data from all the regions that will be using the application. A little bit challenging because regions keep coming back with little changes to their data, but Jim is wrapping it up. Jim accepts that we will have to go live with data that isn't 100% perfectly accurate, but in any case things always get out of date (personnel change, schedules change, etc) so it's not like DCSOps' data was ever going to be static and frozen as of the moment we deploy the realigned version.
Jim knows that we'll need some time to integrate these spreadsheets after he gets them to us. Frank said that he's not too concerned about this. He's looked at the format and knows the data, and he can script most of this with ActiveRecord and Ruby.
-
Rails 6 upgrade
Changes mostly done, but not deployed yet. We are focusing on other items for a bit, and will return to this later, since finishing the upgrade and the related merge will be at least an all-day task. The "related merge" is that the #126 changes and the #185 changes need to be merged together -- a tedious though not hugely complicated task -- since both changes touched large swaths of the code base.
-
3 days with no request-timeout nor memory-exceeded errors on production
The Expedited WAF rules we put into place on Heroku seem to have protected us, at least for the past 72 hours. We'll keep an eye on it.
-
Survey of issues that need further design work done
There's a general issue of having design work remaining to be done in some issues. Frank will do a survey of remaining issues and look for design questions, so that he can ask them up front and get answers even before starting actual work on the issue.
-
Jim and Frank have a call today
NorCen Dispatch Console is on the agenda, and various other topics. Frank mentioned to Karl that he may ask some questions about the "New regional boundary maps" issue.
-
Status of specific issues (from the Work Plan)
-
Status: On staging now.
Note that Frank is not so concerned about incorporating Jim's spreadsheet changes. He can script it with ActiveRecord and Ruby. Jim and Frank have talked about this and Frank thinks it will go smoothly.
-
For "Recent Incidents", display the street name between the city and incident number.
Status: Code complete; merged to master.
-
After clicking "Submit Incident Report", bypass the "Currently Open Incidents" page and go directly to the "Create Incident" page. Also, create link to get to "Currently Open Incidents".
Status: Code complete; merged to master.
-
Paginate the "Currently Open Incidents" page
Status: In progress.
-
In Incident Details, make changes to the "Damage Assessment" section to reflect the Red Cross' current Direct Client Assistance (DCA) model.
Status: Code complete, branch waiting for merge.
-
-
Wait on submitting current hosting invoices
Darlene and Ahmed are working on that still. They need to have it resolved by the end of the week anyway, so Karl will watch his email and be ready to submit the invoices when they give the green light.
-
FY20 realignment work final invoice coming up
Karl will make sure to submit the last invoice for FY20 DCSOps realignment work by Friday the 26th.
-
Getting to Rails 6.
Todd's been working on issue #126, the Rails upgrade issue.
The app has now been brought to Rails 5.2, which is the last release of the Rails 5.x series and should be in stable support for a while.
We are off the
squeellibrary entirely now, and didn't simply migrate tobaby_squeeleither. Instead, the relevant code has just been rewritten to be compatible with both Rails 5 and Rails 6. In Todd's words, this "was more work, but better for the future". This involved changes to both arcdata and to arcdata_core.There are 3 main things left now:
-
Fix the
serialized_columnssituation.arcdata_core has a custom module called serialized_columns -- written by John Laxson for in-house use -- that allows DCSOps to store serialized configuration information to a single DB column, complete with convenience accessors, etc.
But that module is outdated: it's using obsolete ActiveRecord features. Rails probably has better native PostgreSQL
hstoresupport now anyway. We need to either update the module or perhaps even just switch to using native Rails+hstore functionality that is now available.See Todd's follow-up about this in Zulip chat.
-
Get rest of test suite passing with Rails 5.2. It's close, but there are still a few tests failing.
-
Migrate from Rails 5 to Rails 6. Once the above two items are done, this one should be pretty easy.
-
-
Merging changes.
There is a big merge coming, because of the interaction between the issue #126 changes and the issue #185 changes. Both of them involved touching many parts of the code base, so they overlap in a lot of places. Each individual textual conflict site should be easy to resolve, since there is no conceptual conflict between the two change sets, but it's going to be lot of sites.
Frank will do this merge.
-
Deploying latest code to staging for Jim to test.
Frank is upgrading the staging server to get it to where he can push the latest code (with all the issue #185 changes) for Jim to test it out.
-
Smaller FY20 items coming next.
Once the above is done, Frank will be turning his attention to the smaller FY20 items in the Work Plan (smaller in terms of programming time, not necessarily in terms of importance to DCSOps users).
-
Request timeouts in the production instance
Random bots are requesting lots of likely-looking but invalid URLs. DCSOps is spending too much effort each time it says "no such URL" in response, which causes DCSOps to get overloaded and time out requests to real users.
We can tune Expedited WAF so that many of the bot requests are handled at the network edge by Heroku, instead of by DCSOps. Frank has added a new routing rule to our Expedited WAF configuration to cover the specific URLs that the bots have been trying so far.
A Rails upgrade may make request-handling more lightweight, which would solve some more of this problem. However, there's still a minor information leak going on here: when a non-authorized request comes in (i.e., the requester does not accompany the request with any known user credentilas), then DCSOps should unconditionally redirect to the SSO sign-in page, no matter what path was requested. DCSOps shouldn't return an HTTP 404 error (i.e., "no such page"), because DCSOps shouldn't be leaking valid/invalid path information to anonymous bots in the first place.
Attending:
- Ahmed El-Aref
- Darlene Johnson
- Andy Evans
- Steven Meissner
- Jim McGowan
- Karl Fogel
Notes:
-
Rails upgrade (issue #126).
Mostly done. After this call, I had a technical progress check-in with the programmers, and the report from that meeting is that we expect to get to Rails 6. This is even more important to do now that the DCSOps's service lifetime may be further extended.
Note that this ticket got accidentally closed for an hour or so, due to an unexpected interaction between a commit message and GitHub's controversial auto-close feature. It's now open again.
-
Core preparation for realignment is done (issue #185).
-
Jim prepared a realignment spreadsheet template; Frank approved.
Jim will send his sample spreadsheet to Steven.
Jim is collecting information from the regions to create the full spreadsheet set that Frank et al will work with.
-
Volunteer Connection
We have some concerns about about making sure the data export we need from Volunteer Connection goes smoothly. More attention and diligence is needed from both sides.
(After the meeting, Jim arranged a coordination call with the VC administrators, as well as a related one with the SSO team. We'll report back on that at the next regular check-in meeting.)
Jim reiterated that he expects that after we stand DCSOps back up after the realignment interregnum that there will be various minor glitches. They should all be easy to fix, but we should just be prepared to handle them. Morning of Saturday, July 25th is proposed flag day for the realignment.
-
Production instance request timeouts, memory usage issues.
Due to attacks, probably by a bot. Turned on Expedited WAF. We have put some more mitigations in place; see notes from the next meeting for details.
-
Next invoice.
Make sure to invoice before COB June 26th.
-
Pending hosting invoices and current MSA.
Darlene to find the funding source, then Ahmed will turn it into a requisition. Karl will send no current hosting invoices until he hears from Darlene or Ahmed.
-
There are 5 main renames to do. 3 are done, the 4th one (which is the hard one) is mostly done, and Frank and Jim had a conversation to clear up some confusion about the 5th one. The changes touch about 80% of the source files.
Frank expects to be done and have the tests passing by tomorrow morning. There is still a chance of some random misbehaviors in the application, because the rename of "counties" to "shift territories" involved some guesswork: Frank had to infer when a county was really a county and when it was really a shift territory.
The breakage might be things like someone who should have gotten a text won't get a text. Things should work for the most part; we should just keep an eye out for odd behaviors.
There will be a staging environment where Red Cross can poke at things first, before we deploy to production.
-
Rails upgrade (issue #126).
Todd aiming for Rails 5.2 first, with potential for a smooth transition to Rails 6 later (maybe using the automated rails upgrade helper). That second upgrade is not strictly necessary; we will see where we are once we get to 5.2.
We will move to Rails 4.2 with the next deployment push (i.e., of the #185 changes). Note that staging is already on Rails 4.2. To be more complete:
arcdatais on machine stackcedar-14andarcdata-stagingis onheroku-18(which supports Ruby 4.2). -
Deployment policy and branch management.
As noted in CONTRIBUTING.md, there are now branches for the different environments.
The following actions are going to be taken soon-ish for those environments:
- Create production branch
- Merge 4.2 branch into master
- Merge tests branch
- Create candidate-production branch on that sha
- Push candidate-production to heroku-staging
- Merge 185 into candidate production (after completion)
- Push candidate-production to heroku-staging
- Tell Jim that it's up and he can look at it
- Start work on feature branches
- If Jim is not actively testing, then merge small ones into candidate-production and push to stagin
- Keep big ones out
- Merge changes from candidate-production into production
- Push to heroku production
- Repeat until all is done!
-
Go ahead and go from role -> capability
This will eventually include renaming the current names that are more role sounding (like "Region Admin") to capability sounding ('Administer Regions') as that's how the code actually works. This is going to be done by Jim as part of the realignment.
Notification Roles are separate entites and don't need renaming. As an aside, those are the collection of people who get notified when some event occurs
Do change the Roster::RoleScope class, as that links the above Role to some scoping
-
Question about scopes, how do they work?
Jim was unsure how scopes work, and asked Frank to look into it. As an aside, Frank did and it turns out the different roles (capabilities) have different scope meanings DEPENDING on what the capability is.
The scopes are just a list of numbers, and for capabilityes like "county dat admin", the list of numbers link to Shift Territory ids that that person can administer. In the case of "dispatch console", the numbers refer to the ids of
Incidents::Scopeswhich are collections of regions for which that person can do dispatch console things. -
Questions about events
Jim was confused about how there are different event types, and then different events, and how those actually link to something that dcsops changes. These are in the
Incidents::Eventssection.Frank has no idea how these work.
Decision: We will enable Expedited WAF, as discussed on 2020-06-09, but since it involves a DNS change, we'll do it after Frank gets to Colorado, i.e., after tomorrow (Friday).
Progress on issue 185 is good. We guess a June 18th completion date. Karl has updated the Work Plan.
Todd and Frank went through all the Ruby gems (see 2020-06-09 meeting notes). Plan is to get to Rails 5, and then we'll see where we are. Rails upgrades are usually done step-wise anyway. However, where we have to do gem-replacement work, we'll try to do it in a 6-compatible way, to make any later upgrade easier.
The path they discussed in the 2020-06-09 4pm meeting won't work. When an incident arrives, it gets assigned to a response territory (RT) on the spot. Response territories are a DAG; e.g., there's one for Chicago, another for Cook County, etc. Each person has the ability to see certain response territories. Translating their access rights to all the incidents that they can see is done at incident creation time, and we then cache the answer.
While we wanted to convert this to a dynamic lookup, there are probably too many queries and they're too complex. This isn't the way the system has been structured up till now. Answering the question "fetch all the incidents pertinent to a given user" dynamically is not trivial unless you've pre-cached the answer. So much of the FY20 "Administrative Console" work in the Work Plan is not going to happen the way we anticipated. Instead, Jim will hand us a giant spreadsheet, which we'll convert to an idempotent, one-time, rebuild-the-cache script. That script is pretty trivial: just loop over all incidents, running the same routine that gets run on incident creation). This will be how most of the FY20 "Configuration" work in the Work Plan happens. See this discussion for more details.
Karl will talk to Jim about creating that spreadsheet soon.
Todd and Frank are both making extensive ("expansive and vast" in Frank's words) changes. Some conflict resolution is inevitable, but Frank isn't worried about it and anticipates being the one to do it.
Even after the issue 185 work is done, it may behoove us to not to deploy it until the realignment happens. The reason to put it off until realignment day is that in both situations there's a possibility that things would break, and we would prefer them to only break once in the next few months.
Completed audit of gem dependencies. While many support Ruby 2.7, almost all support 2.6 so we should pick the latest patch version of that series. It is well-supported by Rails and Heroku and should be for a while.
There's a similar story in terms of Rails version support (many support 6, but several only go up to 5). Since typical strategy for a large upgrade is to step through each version separately anyway, the plan of action is to take the Rails version to 5 and update all gems that require minimal code changes.
Since Rails 5 is a half-measure, the goal would be to avoid investing
significant time in any Rails 5-specific changes (the largest one being
squeel (Rails 4) to baby_squeel (Rails 5)), where it might make sense to
take another direction (convert to ActiveRecord-based queries) that positions
us for an immediate Rails 6 upgrade.
A VC personnel record comes in carrying the position name and the name of the response territory (e.g., "DAT Supervisor - Greater Chicago". For the ones that match -- i.e., that DCSOps knows about -- DCSOps parses out the position and the geography. (Interestingly, it uses regexps right now for this, instead of doing some kind of table-based match.)
DCSOps's semantics in this process are a little buggy right now. Say someone's personnel data comes carrying two such records: "DAT Supervisor - Greater Chicago" and "Admin - Southern Illinois". Right now, they would become both a DAT Supervisor and an Admin, in both locations. This is not what should happen.
We're going to fix this as part of the overall realignment effort by doing a table-based match, and by providing an administrative UI so a DCSOps administrator can manipulate the information. Since there are three variables to control, and only two dimensions available on the screen, the UI for this needs to be thought through carefully. Jim will sketch out designs for a UI, Frank will give feedback, and we'll iterate until we know what UI to implement.
Looks like our outages are the result of attacks that take advantage of vulnerabilities in our ancient 4.x version of Rails. Each attack seems to come from a single IP address (i.e., DoS but not DDoS).
Frank will turn on Expedited WAF (see add-on page) on Heroku. It's only $75/month. We don't know if we'll need to keep it on after the Rails upgrade, since the upgrade may remove the DoS vulnerability, but we'll figure that out later.
The DCSOps query tool does a kind of screen-scrapy thing to get volunteer information from Volunteer Connection: DCSOps virtually presses the "Excel" download button, basically.
It used to be that each region had a dedicated VC login for this, and there are three different queries, so we were running 3x9=27 queries. But the problem was that sometimes a region admin would shut down the account, not realizing that DCSOps needed it, and then we'd have to scramble and get it reinstated, which involves phone calls and whatnot. So we switched to doing each of the three queries for all the regions at once.
The problem with that is that the first of those all-regions queries is too large. It's a burden on VC. We're pulling all the regions at once, so that's 37,000 records in the first query. VC's feels the load on it transaction table. Diana has asked that DCSOps please stop melting the VC service :-).
During the call, Frank found the code path by which we get data from VC import. (Note: We looked in the Heroku Scheduler to see when this job gets run, but we didn't find it in there at all. So when/how is the cron being run? Karl thinks Frank found the answer to that question, but Karl didn't catch it, so he's putting this parenthetical aside here in the hope that Frank may some day see it and replace it with the answer.)
Here's what we will propose to Diana et al:
Let's have a single admin-level account in VC that has permissions to do whatever queries it needs to do. We will use that account to do smaller queries (say, one per region). To do that, we'll have to add a new query parameter to specify which region/chapter we're asking about, since VC will no longer be able determine that from the identity of the account making the request.
Jim will draft the plan and email it to Frank and Karl for review. Then Jim will send the approved plan to Diana, Frank, and Karl.
Attendees: Darlene, Steve, Ahmed, Karl.
-
Todd Eichel is now working on the Rails upgrade (issue #126). There is a possibility that we go to 5.x instead of 6.x.
Explained about how the main issue is one of deferred maintenance: that some Ruby gems (third-party open source code libraries) were written for Rails 4.x but that since then the community has moved on to some other replacement gem, and the old one does not work in Rails 6 or (in some cases) even in Rails 5. This is mostly what Todd is wrestling with.
-
Realignment work proper (with Frank Duncan) starts this week: (issue #185).
-
Frank is also investigating the production instance memory usage issues.
-
Regarding CSV downloads and data dashboards (BI dashboards)
Three possibilities:
-
Just make a button that does the right query and downloads everything that SiSense needs is easy & a small project.
-
Figuring out what PowerBI would need, and supplying it, would be more of a medium project.
-
Stretch goal: offer schedule dumps in CSV form. This is on "the small side of medium".
After the meeting, Karl sent an email with quotes for above (ref:f0b64ca7):
From: Karl Fogel To: Ahmed El-Aref, Darlene Johnson, Steven Meissner Subject: Notes and quotes. Date: Mon, 08 Jun 2020 15:34:11 -0500 Message-ID: <[email protected]> -
-
MSA is now fully signed.
-
Sent first invoice, which has already been approved.
Note that the invoicing address has now changed to
APCoupa (_AT_) redcross.org. -
How to handle pending hosting/support invoices.
Karl sent the latest DCSOps SOW to Ahmed:
From: Karl Fogel To: Ahmed El-Aref, Darlene Johnson Subject: DCSOps maintenance (hosting/support) SOW. Date: Mon, 08 Jun 2020 15:50:12 -0500 Message-ID: <[email protected]>We'll review and modify as needed, then put it under the new MSA. Once that's done, the pending invoices can be sent.
Re production instance errors: Frank is digging in to the logs. Goal is just to get to a plan for code upgrades that will go out with our next deployment: e.g., if there's additional debugging information that we'd like to have in the logs so that next time we can diagnose more quickly, then this is our chance to learn what that information is and make sure it gets logged.
After that, the actual realignment changes begin (issue #185).
We also recapped the CSV downloads and data dashboards discussion (ref:f0b64ca7).
Frank Duncan and Todd Eichel met and discussed the upgrade. Everything's good so far, however Todd says there is some question as to whether we even can upgrade all the way to Rails 6.x and that we might have to go only to 5.x. We don't know for sure yet. Frank and Todd will check in early next week; we'll know more soon.
TL;DR: Deferred maintenance sucks. This upgrade is a major slog-fest, due to heterogeneity of dependency outdatedness. Many things don't have obvious drop-in replacements in Rails 6; research and rewriting would be required.
None of the work he's doing will be wasted, and the upgrade will get
completed one way or another, but we may need to finish it after the
One Red Cross realignment instead of before. This is an acceptable
scenario, especially given Heroku's delaying of their cedar-14
deadline, but it wouldn't be our first choice.
For now, Frank will continue on the time-boxed path he's on, and we'll see how it looks in a few days.
-
Karl will send the first invoice to the official IT procurement address, CC'ing Ahmed, Darlene, Steven, and Latish.
-
Karl to discuss PowerBI and data dashboards with Jim, estimate costs of the various options.
(Some of this has now happened, after this check-in meeting. Frank talked to Jim at 5pm CT today and raised this topic at Karl's request. Then later Frank and Karl estimated the various options. Karl will discuss further with Jim, but can give the estimates at the next meeting.)
-
We discussed current status, which is mainly the Rails upgrade. It's ugly, but only to the expected degree, and things are on track. (Also: I can't believe I just unintentially used the expression "on track" in referring to an upgrade of software named "Rails".)
-
MSA: Darlene and Ahmed talked, and now Ahmed is circling back with OGC and Risk Management. Note that pending and future hosting invoices may get rolled into the MSA. Meanwhile Rod is checking about past invoices.
DCSOps is currently on Rails 4.1 and uses the now-obsolete Squeel for query generation. Squeel is no longer maintained and only works up to Rails 4.x. Its successor is BabySqueel, which does work with modern Rails 6.x, but it's not a drop-in replacement -- you have to migrate your code from Squeel syntax to BabySqueel syntax, which is doable but certainly not trivial.
Given that this was the very first Rails 4->6 issue Frank ran into, he wanted to know how worried we should be, and whether or how much he should time-box the upgrade to 6. We discussed it and Karl concluded that this is probably one of the harder issues in the upgrade and that Frank just happened to run into first. Frank will continue with the upgrade to 6 (and thus with the Squeel->BabySqueel transition). We will check in about the overall upgrade on the morning of Friday, June 5th.
Regarding the recent "critical memory errors" that have been happening on the production box, Karl will download the logs to bigdata (note: this is now done, in bigdata:r80), to make it convenient for Frank to dive in and look at them as soon as he can. Karl will also take a look at them. Also, Karl didn't mention this in the meeting, but he's going to sign up for Heroku alerts, as Jim already has. (Also done, at 5:54pm CT.)
We expect future costs to be roughly the same as current costs:
- Heroku (app dynos plus add-on services): appx $160/month
- Outbound email (PostmarkApp now, Sendgrid later): appx $80/month
- Twilio text messaging API: appx $50/month
- DNSimple (domain registration and related services): appx $30/month
Some of those DNSimple costs (the $27/month they charge for "domain professional services" may be unnecessary; Jim is going to look into that).
We don't expect the PostMark -> SendGrid transition to affect the cost much. (SendGrid is part of Twilio; Jim has spoken with executives there about this transition.)
We'd had some concern that we might need to start paying for Papertrail, but further investigation showed that the log-limit excession was a temporary event and that normally we stay below 65 MB. Frank and Karl turned on flood protection, and we could turn on log entry filtering too if we ever need to (but we probably won't need to).
Regarding SiSense (formerly Periscope), the BI/dashboard service through which Jim extracts data reports:
At first it was free (the original developer, John, did a logo access trade, though that wasn't officially approved and ended quickly). Then it was $100/year, which was basically them being nice to the American Red Cross. But they eventually wanted the full price, which is much higher: $2k/month. This is comparable to competitors like, say, Tableau Server, which is $25k/year.
Really, Jim just needs a tool by which he can get incident data easily out of DCSOps. As a bonus, if it were a tool that other authorized people could use, that would be great. Best solution is just for the Search Results page (e.g., https://dcsops.org/incidents/cni/incidents for CNI, etc) to have a "Download Data" button (that appears only for properly authorized users) that just downloads a CSV file.
OTS will come up with a quote for this, making sure to include the various filtering features and other unforseen things that Jim and the region administrators will undoubtedly think of. Bonus: this would enable authorized people other than Jim to get data out of DCSOps.
On or about July 13th, we'll stop the doing imports from VC to DCSOps. Then all the regions do all their reconfiguring, and then we turn on the (newly re-aligned) DCSOps and it starts doing imports, and we watch carefully and make sure it all goes right.
Since Fiscal Year starts July 1, though, we need to make sure that incident data captured from July 1 -> July 20th (or whenever we turn it back on), we see that incident data in its correct new places. Also, ability to front-load schedules for, say, the first 72 hours.
Karl has sent an email to Jim and Frank to request that they start the technical conversation about this.
-
Karl clarified that the period beyond June 11th in the Work Plan will be filled in with more detail before we get there.
-
Karl has updated the Work Plan to indicate that the "Rails upgrade" item is issue #126 (as well as being related to issue #157).
-
Jim mentioned that we'll have a flag day on which DCSOps does its switchover to the new Volunteer Connection all at once.
There will be an interim period, likely lasting about a week, during which DCSOps doesn't do its regular data pulls from VC because VC will be on the new One Red Cross while DCSOps isn't yet. That's okay: DCSOps will still continue working with data it had as of its most recent synchronization with VC, and region administrators have been apprised of this. Basically, DCSOps will temporarily still have an "old view" of the world during this brief interim period.
-
Discussion of various procurement/budgeting/invoicing matters:
-
Send FY20 invoices on June 1, June 15, and appx June 30 (the main thing is, make sure they're all sent in June).
-
In each invoice, explain which SOW items it covers.
-
For FY21 work that is undertaken early, indicate that in the invoice. There's no need to attach a dollar amount to it; just call it out so that we can keep track of early FY21 progress.
-
Darlene will check with Ahmed regarding when OTS should send the most recent (new) DCSOps monthly hosting service invoices.
-
Darlene will check with Ahmed regarding the FY21 DCSOps hosting service agreement. (Ahmed is working with OGC to set up an MSA.)
-
Jim and Karl will determine the monthly costs for DCSOps in FY21. Those costs will be similar to the costs in FY20 and previous years, but there may be some minor changes due to new use of Sendgrid, the data intelligence/analysis service Jim uses (whose name Karl forgot, but Jim can remind him and then he can update this part of the notes), and possibly greater log storage costs at Papertrail, though Karl is looking into that to see if it's necessary.
-
Test coverage is far from complete right now. The incidents code has some coverage via feature tests; the scheduling code has basically none. Of the 6500 lines of Ruby in the application, there are now 3100 or so lines covered by feature tests, and all the unit tests together cover 4400 lines.
(For context, the three main things in DCSOps are: Admin, Scheduling, Incidents.)
A Rails upgrade is medium risky right now. The app will likely remain usable and downtime due to the upgrade should be low (assuming we've done the manual smoke tests). People will be able to create incidents and do common operations. But there will likely problems in code that is run less often, say, something like the VC connection importer that runs twice a day. Or, for another example, NY has extra business logic written in to the app, enabled through some special configuration changes. If that were to break in the upgrade, we would probably only find out about it after the fact.
But time-wise, we can't afford to make the test suite perfect before doing the Rails upgrade, followed by "The Big 185", followed by the actual realignment changes.
Therefore, we'll commence the actual work but continue adding tests as a background activity (roughly one or two new tests a day). We won't get perfect test coverage this way, but we'll at least reduce the risk-per-change. Note that the test-writing process up till now has been about learning, so it's been serving a double purpose. It will continue to help with learning, but less and less so as we understand more and more.
-
Frank to start on #126 (Rails upgrade). Will upgrade to latest Rails as of today. Also will clean up related issues and pull requests (e.g., #157, #184.
-
Frank to start on The Big #185. See also Frank's email "Region migration information and estimate" from July:
From: Frank Duncan To: Karl Fogel Subject: Region Migration Information and Estimate Date: Wed, 3 Jul 2019 10:25:06 -0500 Message-ID: <CAK=Dgqf6OZMP0mp6eZ+UYvkCWXD9M0nTXfim6n_E6cbgKDnCuA@mail.gmail.com> -
Frank to upgrade our staging server to master. (It may have occasional small changes that are supersets of master, when we're testing something, but the norm will be that staging doesn't diverge far from master.) We will point this staging server to either Prod SSO or Staging SSO as needed.
-
Karl to set up Meeting Notes area (done -- that's here).
-
Frank and Karl made sure Frank has access to the private notes area. Sometimes an issue may involve PII or other sensitive data, and in those cases, we use the private notes area, although these public notes may mention the matter of course.
-
Karl to work on getting us test accounts in ARC SSO.
-
Karl has confirmed start of work to ARC HQ.
-
Karl has let Jim know that Frank would like to call Jim, and given Jim Frank's number.
-
Frank to get smoke-test instructions from Jim.
-
Karl to get Frank the list of what's included in the SOW. DONE: See the Work Plan.