You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
❌ there is no "i" button except for the main one next to the login name
Confirm that no Tales are running
❌ How am I supposed to do that?
Search
Enter "Ligo" as search term. Confirm only LIGO tale displays
Clear search term. Confirm all tales display
Filter
Navigate to the Browse page
Confirm that you can switch between "All Tales" and "My Tales"
Confirm that all public Tales are shown on "All Tales"
Confirm that your Tales are shown under "My Tales"
Select "My Tales"
View a Tale and then navigate back to Browse
Confirm the filter is reset
❌ this is the wrong behavior. If I go back, I expect to see whatever was there last.
View tale
Select "View" button on LIGO Tale
Confirm that Run > Metadata page displays
❌ Run > Interact is what I see when I click "View" on a tale
Confirm that you cannot delete or edit Tale properties
❌ I can change the environment, license, and description, but I cannot save the changes
Select Back button. Confirm that you are taken back to Browse page
Select "View" button on Tale you own
Confirm that the Run and Close buttons are present
⭐ They were there for the LIGO Tale, too
Confirm that you can edit Tale properties
❌ If I click "Save", it complains about a missing author, which it shouldn't because I am listed as the author
❌ "Save" does not give me any feedback. It should probably be insensitive if no edits have been made.
❌ Should an ORCID ID be required here?
Select "Close"
Confirm that you are taken back to Browse
Launch instance
Select "Run Tale" button on "Ligo" tale
Confirm you are prompted to Copy and Run
❌ Running is not a mutative operation in my opinion. It should not require copying.
❌ More importantly, the current implementation does not allow me to run the code in a tale without making a copy. Once I made a copy, I have no guarantees that the tale is precisely the same as the original tale. This basically means that I have no means of guaranteeing that an experiment is reproducible.
Select Copy and Run
Confirm that you are redirected to the Tale's Run page
Confirm that the notification dialog appears
❌ I did not see a notification dialog.
Confirm that the step progression is correct
❌ don't know what this means.
Select "Stop" to end the Tale instance
Confirm that the Tale has stopped
The content of the "interact" tab disappears immediately, but I have no other means to confirm that the tale has stopped and, presumably, it hasn't until the "Stop" button has changed to "Run".
Confirm that the button has changed to Run
Navigate to Browse and select the "Run Tale" button on a Tale you own
⭐ The "Run Tale" button changes to a spinner. I would rather see an overlay on top of a disabled button.
⭐ More fine-grained progress would be nice here
❌ Half an hour later, it's still spinning, so something went wrong here, but I have no idea what.
❌ Re-did this and the tale launched and I can see the workspace and the "Stop" button is enabled. However, if I click on "Browse", the "Run Tale" button still shows a spinner. When I refresh the "Browse" page, "Stop Tale" is correctly shown.
After the instance is launched, try to delete the Tale (not instance)
Confirm you cannot select the delete button on the browse page
Confirm that the "Run Tale" button has been replaced with "Stop Tale"
Manage
Preconditions
Empty home directory
⭐ Deleting a file does not prompt for confirmation, nor does there seem to be a way to recover a deleted file or undo the delete operation. This is not nice.
No registered data
Register General
Select Manage > Data > "+" to open register panel
Confirm list of providers matches list of supported providers (Globus, DataONE, Dataverse)
⭐ There is text below the search box, but it appears static. If "list of providers" above refers to that seemingly static text, then yes.
Register DataONE data
Select Manage > Data tab
Select "+". Confirm "Search for Data Sources" modal displays
Search for DOI 10.5065/D6862DM8
Confirm "Humans and Hydrology" dataset is found on DataONE.
⭐ the folder name entry box could be wider
Select Register
Confirm dataset appears in Manage Data list
⭐ There is no indication at first that anything is happening. There should be. Perhaps the dataset should be shown greyed-out with a note/progress bar that indicates that it is being imported
Select the Create New... button on the Browse Page
Select the Create New Tale dropdown item
Enter title "Test Tale"
Select compute environment RStudio
Select "Create New Tale and Launch"
Confirm notification dialog opens
❌ I do not see a notification dialog. After clicking "Create New Tale and Launch", there is a delay without feedback, after which the browser is redirected to the "Interact" tab for the tale.
Confirm that the build/launch steps are correct
Confirm that you are redirected to the Run page
Compose Jupyter Tale
❌ "Jupyter with Spark" results in endlessly launching tale that I cannot stop or delete. This is bad because it counts against my running tales quota.
Compose OpenRefine Tale
Too many instances
Make sure you have two launched Tales
Attempt to create a Tale
Confirm that you receive an error that you have the max limit
⭐ I get a message box that says "There was an error while creating your Tale". This is fine, but it should tell me what the error is.
Confirm that the Tale is created, but not the instance
Run
Preconditions:
No running Tale instances
❌ can't fully meet this since I still have the straggler instance from above
General
Launch (Copy and Run) the Ligo Tale and wait for instance to start
⭐ we need some sort of systematic transition feedback for operations that aren't instant (i.e. require round-trips to the server).
Confirm you are on the Run page
❌ Unfortunately, the LIGO tale is also stuck in starting mode.
❌ Ah, after about an hour of LIGO tale starting and a few hours of Jupyter+Spark starting, I got notifications informing me that the starting on these have failed. The notifications do not have timestamps, so I cannot tell when the failures occurred. There are no timestamps in the logs either.
❌ However, even after refreshing the page, the LIGO and J+S tales are still shown as starting, so I cannot launch new tales.
Confirm Run panel displays Tale icon, title, and author
❌ It list me as an author, which is wrong. I am the owner of this instance, but I have not authored a single bit in this tale.
Confirm Run page displays Interact, Files, Metadata tabs
⭐ Everything in that menu pertains to the current tale, except the docs, which are for tales in general.
Confirm fullscreen displays Run panel in full screen
⭐ it does, but the menu does not close when I click on "Fullscreen" or the export options.
Confirm that the options for exporting the Tale are present
Interact tab
Select the "Interact" tab
Confirm Jupyter environment is displayed in iframe
Select "popout" icon. Confirm Jupyter is opened in new Tab
Files tab
Select "Files" tab
You should see a vertical navigation menu with three tabs: "Home", "External Data", and "Tale Workspace".
Metadata tab - Read Only Tale
View (not launch) the LIGO Tale
Navigate to Run
⭐ Not sure what this means
Select "Metadata" tab
Confirm that metadata displays for current tale
Confirm that the form elements are read only
❌ "+ Add an author" should not be there or it should appear disabled. That said, it does not do anything.
Metadata tab - Owned Tale
Launch a Tale that you own
Add multiple datasets that have DOIs to the Tale
❌ Can't do this since the tale is still starting. To be more specific, I can click the "+" button and select a dataset, but after a few minutes, it is not showing under "External Data"
Navigate to Run > metadata
Valid environments should be listed in the Environment dropdown menu
Valid licenses should be listed in the License dropdown
Confirm that you see Created by <your_name> under Authors
Confirm that your Tale has the CC4 license assigned to it
Confirm that the Tale's Published Location reads This Tale has not been published
Confirm that there are citations for the external datasets
❌ see above
Change the name of the Tale, Category, Description, Illustration, Public, License, and add multiple authors
❌ Adding an author should not replace the existing author.
Select Save. Confirm changes are saved.
Refresh the page. Your changes should be preserved between page refreshes, confirming that they are being properly persisted.
Confirm that the new Tale authors are displayed on the Tale card
Home
Select Files > Home. Confirm that your home directory displays
Create a folder and upload a file to the folder
⭐ Using the "+" next to "Home" to create a directory in a directory is not quite intuitive.
⭐ Upload notification in yet a different place and styling from other notifications
Precondition: Register one or more Datasets (e.g. doi:10.18739/A29G5GD0V) on Manage > Data (see Manage > Register test cases above)
NOTE: You may need to wait for registration to complete
Confirm "External Data" list is initially empty
⭐ should probably specify that this refers to the "Files" tab in a tale.
Confirm (+) button ("Select Data modal") behavior
Confirm that "WholeTale Catalog" shows all registered datasets
Confirm that "My Data" filters to only show your registered datasets
"Add" data to associate it with a Tale
⭐ The interface is inconsistent with "Manage". In "Manage" the "+" is next to "Home", etc., whereas in a tale, it's on top of the listing, which is probably better since we add and remove things from the listing.
"Remove" data to dissociate it with a Tale
❌ I only have the option to download a dataset. After some poking, it turns out that I can remove datasets by clicking "+", but that's not nice.
"Select" saves your changes back to the server
"Cancel" aborts the operation without saving and reverts your changes
Confirm "External Data" list matches what was saved
Workspaces
⭐ the "+" button should not be context dependent when that context is far from it visually. In other words, "+" for "External Data" should also display a menu. It should also maybe have a different icon, one consistent with either "Home/Workspace Options" or "Context Menu".
Confirm create folder
Confirm rename folder/file
Confirm upload file
Confirm copy file/folder from/to Home
❌ The file context menu is inconsistent between "Home", "External Data" and "Workspace". I don't quite understand why "Copy" and "Move To" are only available in Home.
❌ I can't copy things from Home to Workspace. There is no such option.
❌ When I copy a file in Home, the (1) comes after the extension instead of before.
❌ When I try to download a file with (1) after the extension, I'm prompted to save it as a ".zip".
❌ The second time I copy a file, it results in a file with zero size. If I try to download it, I get an empty .zip file.
❌ If I download a renamed file, the file name is some weird version of the name before renaming.
Confirm move file/folder from/to Home
❌ Move in "Home" does not allow me to move to Workspace. There is no "Move" on items in workspace.
Confirm remove file/folder
Select "+" button has options "New folder", "Upload file", Import Tale Data..."
Folder operations
Confirm context menu Move To, Rename, Copy to Home, Download, Remove
❌ There is no "Move To", but I see this may be intentional.
Move To...
Issue -- should this be enabled
Confirm rename folder
Copy to Home confirm folder copied to home
Confirm download -- zip file
Confirm remove
File operations
Upload a file
Confirm context menu Move To, Rename, Copy to Home, Download, Remove
Move To... Home?
Issue -- doesn't work
Confirm rename folder
Copy to Home confirm folder copied to home
Confirm download
Confirm remove
Select "Select Data..."
Note vertical navigation with Home and Tale Workspaces
⭐ I'm not sure what this means
Copy to Workspace from Home
Move to Workspace from Home
Copy to Workspace from another workspace
Move to workspace from another workspace
Exporting - ZIP
View a Tale that you own
Navigate to Run
Click the ... dropdown
Select Export as Zip
Confirm that you are asked to start a download for the archive
Open the zip archive and confirm that the top level has
LICENSE
README.md
metadata/
Navigate into the metadata folder
Confirm that there is a manifest.json and environment.json file
⭐ environment.json contains an icon, which is a reference to a file in the master branch of a github repo for jupyter. This may or may not work as expected in the future, depending on whether the jupyter project maintains or changes the file path and its contents in master. We should probably think about what we want the behavior here to be.
Exporting - BagIt
View a Tale that you own
Navigate to Run
Click the three-dot-dropdown
Select 'Export as BagIt'
Confirm that you are asked to start a download for the archive
Open the archive
Confirm that the top level has
run-local.sh
tagmanifest-sha256.txt
tagmanifest-md5.txt
manifest-sha256.txt
manifest-md5.txt
bag-info.txt
⭐ Another reference to a master branch in github. I think we should at least give some thought to the idea that this archive necessarily conforms to all future versions of that file. Alternatively, we should point to the specific instance of that file in time used to create the archive.
README.md
bagit.txt
fetch.txt
data/
metadata/
Navigate into the metadata folder
Confirm that there is a manifest.json and environment.json file
Navigate to /data
Confirm that there is a LICENSE file
Analyze in Whole Tale
Task 1: Importing a Dataset from Dataverse
These test cases cover potential situations that can occur when importing datasets from Dataverse.
Confirm that the Tale title reads as Replication Data for: "Agricultural Fires and Health at Birth"
Confirm that the only item in the Selected data section matches the uri with Data Source appended
Confirm that no environment is selected
Confirm that the Create New Tale and Launch button is disabled
Select an environment
Click Create New Tale and Launch
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists in the Tale
* ❌ The tale failed to start with no actual error in the log. However, I see the "Stop" button being active.
* ⭐ When I launched this tale, the progress was shown as notification. I think notifications should be for background tasks. It seems silly to have the most relevant information related to what I am doing in a corner, while most of the page space displays a useless spinner on a big, equally useless, logo.
Task 2: Importing a Dataset from DataONE
These test cases cover different scenarios that may happen when importing datasets from DataONE.
Case 1: Ideal Scenario:
This test covers the ideal case where dataset importing has all of the needed information
to properly run. This includes the dataset location, title, and selected environment.
Confirm that the Tale title reads as Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, Russia
Confirm that the only item in the Selected data section matches the uri with Data Source appended
Confirm that Rstudio is selected in the Environments widget
Confirm that the Create New Tale and Launch button is enabled
Click Create New Tale and Launch
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists in the Tale
Task 3: Shared Behavior
These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.
❌ Case 1: Missing Data Location Query Param:
The test covers the case where the dataset location is missing as a query parameter. This query parameter is used to let the UI know
that we're importing a dataset/Tale. If it is missing, the default Compose page should be observed.
Confirm that the Tale title reads as https://dataverse.harvard.edu/api/access/datafile/3323458
Confirm that the only item in the Selected data section matches the uri with Data Source appended
Confirm that no Environment is selected
Confirm that the Create New Tale and Launch button is disabled
Select an environment
Click Create New Tale and Launch
Confirm that the notification bar appears & properly updates
Confirm that you are redirected to the run page
Confirm that the Tale name matches the Tale Name in the Create Tale Modal
Confirm that the data exists in the Tale
* ⭐ It has a proper name though, so the query parameter is uses only in the initial dialog. I'd argue that is unnecessary. If the URI points to a dataset and the dataset has a name, and the user does not override that, there is no good reason that I can think of not to use it.
Tale metadata tests
The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.
manifest.json
Launch a Tale that you own
Add a dataset to the Tale
Add at least one author
Add at least one file to the workspace
Export the Tale (as ZIP)
Open the manifest.json file under metadata/
Confirm that the author is listed under schema:author
Confirm that you are listed under createdBy
Confirm that the file(s) in the workspace are properly listed in aggregates
Confirm that the external dataset is referenced in aggregates
Confirm that the dataset is also listed under Datasets
environment.json
Open the archive from the manifest.json test
Navigate to metadata/environment.json
Open the girder web API
Locate and GET the Image that the Tale used
Confirm that the config section from girder matches the config section in environment.json
LICENSE
Open the archive from the manifest.json test
Open the LICENSE file
Navigate to Run > metadata
Confirm that the opened LICENSE file matches the selected license on the metadata page
Change the Tale's license
Export the Tale
Confirm that the LICENSE changed appropriately
Publishing tests
Case 1: Un-published Tale
Launch a Tale that you own
Add data to your Tale-this can be to the workspace or as external data
Click the ellipsis dropdown on the Run page
Click Publish tale...
Confirm the the links in the info icons open in new tabs and resolve
Confirm that you can select production and test DataONE servers
Select the DataONE Development repository
⭐ Why is the dropdown list expanded here and not in other places?
Click Publish
Confirm that you are presented with a popup asking you to login to DataONE
Click Continue to DataONE Login
Log into ORCID
Confirm you are redirected back to Whole Tale
Confirm the publishing modal automatically opens
Reselect DataONE Development and click Publish again
Confirm that the progress bar begins and disappears once complete
Confirm that an identifier is shown in the modal dialog.
Click the identifier
Confirm that you are brought to DataONE in a new tab
Confirm that the package lists
metadata.xml
LICENSE
README.md
manifest.json
environment.json
Confirm that the Data Set Creators section lists your account
❌ It lists my fake author that I set in the metadata page on WT. It does NOT match the creator (my WT account)
⭐ The ORCID account I used to publish the tale is not shown in any of the entries in "People and Associated Parties".
Confirm that the Data Set Usage Rights matches the license on the Tale's Run > metadata page
Confirm that the Published Location on the Run > metadata page is correct
Case 2: Re-published Tale
Launch a Tale that has been published before
Take note of the Published Location in Run > metadata
Re-publish the Tale to the DataONE Development repository
❌ I got some errors at first:
TypeError: Cannot read property 'coordinatingNode' of undefined
at o.submitPublish (wholetale-3d657eb9ab01fce6c889d75c44a3d276.js:104)
at o.send (vendor-bbd084e1512157a0832af7f993f36555.js:3232)
at o.n [as send] (vendor-bbd084e1512157a0832af7f993f36555.js:3197)
at vendor-bbd084e1512157a0832af7f993f36555.js:2197
at e.flaggedInstrument (vendor-bbd084e1512157a0832af7f993f36555.js:2496)
at vendor-bbd084e1512157a0832af7f993f36555.js:2197
at e._run (vendor-bbd084e1512157a0832af7f993f36555.js:1712)
at e._join (vendor-bbd084e1512157a0832af7f993f36555.js:1711)
at e.join (vendor-bbd084e1512157a0832af7f993f36555.js:1680)
at Function.y.join (vendor-bbd084e1512157a0832af7f993f36555.js:2316)
Ember.onerror @ wholetale-3d657eb9ab01fce6c889d75c44a3d276.js:38
After some repeated clicking on "Publish" and "Cancel" it worked.
❌ That said, "Cancel" still does not close the publish modal.
Confirm that the Published Location has changed to the correct package landing page
⭐ Maybe we want to think about this some more. If I modify a tale that was published it (the modified version) is, arguably, not published any more. If I re-publish a tale without modification, the old published version does not go away. WT can pretend it's not there, but that's not a correct reflection of reality. It is also possible to publish the same tale in multiple places, so we should be able to accommodate that idea (this may involve some sub-discussion; for a paper, this would be rare, since so much of the academic world shallowly equates "published instance" with "unit of research", so publishing the same paper twice is seen as gaming the system, although I'd argue it's the system that's probably silly; anyway, not so for artifacts like code and data, since everybody accepts that "unit of research" there consists of the substance rather than instances of pointers to it)
Open a non-incognito browser tab, log into the UI and navigate to the /manage view
Create a folder (e.g. test123) and click down into it
⭐ Maybe pressing "Enter" should be equivalent to clicking the default button
Open an incognito browser tab, login to the UI again, and navigate to the /manage view
In the incognito tab, delete the test123 folder and close the incognito tab
Refresh your non-incognito browser tab
You should not be stuck on a blue screen
You should not need to manually refresh the page
❌ I'm not sure I understand. What does "Refresh your non-incognito browser tab" refer to? If it's something I need to do, it's arguably a manual process, so it seems to contradict this point.
You should be taken to the /manage view of your Home folder
NOTE: Your last refresh may take a couple of seconds longer than usual
Test Plan
Note: For all tests, repeat for supported browser/OS combinations.
Preconditions:
Authentication
Navigation
Browse
Preconditions:
Assumes production Tales present (e.g., LIGO, materials, etc).
No running instances
General
Manage
Preconditions
Empty home directory
No registered data
Register General
10.5065/D6862DM8Tale Creation
Compose Jupyter Tale
❌ "Jupyter with Spark" results in endlessly launching tale that I cannot stop or delete. This is bad because it counts against my running tales quota.
Compose OpenRefine Tale
Too many instances
Run
Preconditions:
No running Tale instances
General
Environmentdropdown menuLicensedropdownCreated by <your_name>underAuthorsPublished LocationreadsThis Tale has not been publisheddoi:10.18739/A29G5GD0V) on Manage > Data (see Manage > Register test cases above)...dropdownExport as ZipLICENSEREADME.mdmetadata/metadatafoldermanifest.jsonandenvironment.jsonfilerun-local.shtagmanifest-sha256.txttagmanifest-md5.txtmanifest-sha256.txtmanifest-md5.txtbag-info.txtREADME.mdbagit.txtfetch.txtdata/metadata/metadatafoldermanifest.jsonandenvironment.jsonfileLICENSEfileAnalyze in Whole Tale
Task 1: Importing a Dataset from Dataverse
These test cases cover potential situations that can occur when importing datasets from Dataverse.
Testing Steps:
Replication Data for: "Agricultural Fires and Health at Birth"Selected datasection matches the uri withData SourceappendedCreate New Tale and Launchbutton is disabledCreate New Tale and Launch* ❌ The tale failed to start with no actual error in the log. However, I see the "Stop" button being active.
* ⭐ When I launched this tale, the progress was shown as notification. I think notifications should be for background tasks. It seems silly to have the most relevant information related to what I am doing in a corner, while most of the page space displays a useless spinner on a big, equally useless, logo.
Task 2: Importing a Dataset from DataONE
These test cases cover different scenarios that may happen when importing datasets from DataONE.
This test covers the ideal case where dataset importing has all of the needed information
to properly run. This includes the dataset location, title, and selected environment.
Testing Steps:
Fire influences on forest recovery and associated climate feedbacks in Siberian Larch Forests, RussiaSelected datasection matches the uri withData SourceappendedRstudiois selected in the Environments widgetCreate New Tale and Launchbutton is enabledCreate New Tale and LaunchTask 3: Shared Behavior
These cases show errors that are common between Dataverse and DataONE. Although they are tested using DataONE URIs, the errors should be the same when replacing it with a Dataverse URI.
The test covers the case where the dataset location is missing as a query parameter. This query parameter is used to let the UI know
that we're importing a dataset/Tale. If it is missing, the default Compose page should be observed.
Testing Steps:
Selected datasection is blankCreate New Tale and Launchbutton is disabledThis test covers the case where the name query parameter is missing. The expected behavior is
that the uri query parameter is used instead.
Testing Steps:
https://dataverse.harvard.edu/api/access/datafile/3323458Selected datasection matches the uri withData SourceappendedCreate New Tale and Launchbutton is disabledCreate New Tale and Launch* ⭐ It has a proper name though, so the query parameter is uses only in the initial dialog. I'd argue that is unnecessary. If the URI points to a dataset and the dataset has a name, and the user does not override that, there is no good reason that I can think of not to use it.
Tale metadata tests
The purpose of these tests are to confirm that the metadata files (manifest.json, environment.json, LICENSE) we generate are correct.
manifest.jsonfile undermetadata/schema:authorcreatedByaggregatesaggregatesDatasetsmetadata/environment.jsonconfigsection from girder matches theconfigsection inenvironment.jsonLICENSEfileLICENSEfile matches the selected license on the metadata pageLICENSEchanged appropriatelyPublishing tests
Case 1: Un-published Tale
Publish tale...DataONE DevelopmentrepositoryPublishContinue to DataONE LoginDataONE Developmentand clickPublishagainmetadata.xmlLICENSEREADME.mdmanifest.jsonenvironment.jsonData Set Creatorssection lists your accountData Set Usage Rightsmatches the license on the Tale's Run > metadata pagePublished Locationon the Run > metadata page is correctCase 2: Re-published Tale
Published Locationin Run > metadataPublished Locationhas changed to the correct package landing pageRegression tests
/manageviewtest123) and click down into it/manageviewtest123folder and close the incognito tab/manageview of your Home folder/run/ThisIsObviouslyNotARealID/browsepage insteadThe text was updated successfully, but these errors were encountered: