[3.3] 1924 export associations as resource relationship #1933

Atticus29 · 2024-12-04T18:25:39Z

Description

This PR allows for the download of related records. It addresses issue #1924 .

From the issue specs:

Adds DwcArchiverResourceRelationship class (following format similar to DwcArchiverIdentifier, DwcArchiverMaterialSample, and DwcArchiverAttribute
Add include Resource Relationship extension checkbox to all download interfaces and web services
Currently External and Internal Associations are concatenated into associatedOccurrences within occurrences.csv core file.
associatedOccurrences will be removed from DwC-Archive export (simply comment out lines 137 and 138 within DwcArchiverOccurrence). We might remove this column from the omoccurrences schema, but let's just comment it out for now.
Within DwcArchiverOccurrence getAssociationStr and getAssociationJSON will be completely removed.
appendSpecimenDuplicateAssociations function will be removed with functionality integrated as a batch function, similar to functions called between lines 1821 and 1827 within DwcArchiverCore
Currently observation associations are appended into the associatedTaxa field, which needs to be removed. The associatedTaxa field will remain in the DwC Archive export, but should now only contain verbatim text from the associatedTaxa field.

QA Notes

Live on dev001. In addition to all of the nuances of what we should map from Symbiota Native fields to DwC, note that I haven't tested the Darwin Core-only download or skipping the download entirely in a while.

Notes/Known Issues

This PR does not accommodate external or observational associations. They are in a separate issue.

Pull Request Checklist:

Pre-Approval

Post-Approval

Thanks for contributing and keeping it clean!

…r method

…nd begin to implement some of the export logic

…sociation search

themerekat · 2024-12-04T18:41:56Z

Change verbiage here from "include Associated Records" to "include Resource Relationships (associations & linked resources)"

…ing used

Atticus29 · 2024-12-10T00:37:15Z

.vscode/launch.json

+      "request": "launch",
+      "port": 9003,
+      "xdebugSettings": {


This file was autoformatted. But also the xdebugSettings section was added because my xDebugger values were being very confusingly truncated. Fixed here slash not at all sure why this is being tracked in the repo.

…etAssociationStr

Atticus29 · 2024-12-10T01:11:40Z

include Resource Relationships (associations & linked resources)

Fixed in 20a3b08.

…traints

…n twice

… omoccurrences so that occurrenceID can be queried

- Load associations via 2 separate queries: -- one using occid -- the other using occidAssociate with the relationship translated to its inverse

association export mod

…tifiers

…wnload

…s recommendation

…ogic

…rceID, which required some logic change since omoccurrences does not have objectID nor instanceID

Atticus29 · 2025-01-22T20:11:06Z

classes/DwcArchiverResourceRelationship.php

+		$termArr['deleteMeOccidAssociate'] = 'https://dwc.tdwg.org/terms/#dwc:resourceRelationshipID';
+		$columnArr['deleteMeOccidAssociate'] = 'oa.occidAssociate';
+		$termArr['resourceRelationshipID'] = 'https://dwc.tdwg.org/terms/#dwc:resourceRelationshipID';
+		// $columnArr['resourceRelationshipID'] = 'IFNULL(IFNULL(IFNULL(oa.objectID, o.occurrenceID), o.recordID), oa.resourceUrl)';


I'll delete some of these commented-out lines once we're more confident about the current strategy.

Atticus29 · 2025-01-22T20:11:34Z

classes/DwcArchiverResourceRelationship.php

+		return array_diff_key($dataArr, array_flip($trimArr));
+	}
+
+	// @TODO decide if setDynamicFields is needed


I don't think so, right?

Atticus29 · 2025-01-22T20:13:30Z

classes/DwcArchiverResourceRelationship.php

+				foreach($this->fieldArr['fields'] as $colName){
+					if($colName) $sqlFrag .= ', ' . $colName;
+				}
+				// $this->sqlBase = 'SELECT ' . trim($sqlFrag, ', ') . ' FROM omoccurassociations ';


Will remove upon approval.

Atticus29 · 2025-01-22T20:15:28Z

collections/admin/importextended.php

 							</div>
 							<div class="formField-div">
 								<input name="collid" type="hidden" value="<?= $collid ?>" >
 								<input name="MAX_FILE_SIZE" type="hidden" value="10000000" />
-								<button name="submitAction" type="submit" value="initiateImport"><?= $LANG['INITIALIZE_IMPORT'] ?></button>
+								<button name="submitAction" class="submit__button--no-left-margin" type="submit" value="initiateImport"><?= $LANG['INITIALIZE_IMPORT'] ?></button>


Little styling fix

Atticus29 · 2025-01-22T20:15:46Z

collections/datasets/datapublisher.php

@@ -8,7 +8,7 @@
 else include_once($SERVER_ROOT . '/content/lang/collections/datasets/datapublisher.en.php');
 header('Content-Type: text/html; charset=' . $CHARSET);

-$collid = array_key_exists('collid', $_REQUEST) ? filter_var($_REQUEST['collid'], FILTER_SANITIZE_NUMBER_INT) : 0;
+$collid = array_key_exists('collid', $_REQUEST) ? filter_var($_REQUEST['collid'], FILTER_SANITIZE_NUMBER_INT) : 0; // @TODO collid is really coming from db in the searchvar attribute of the request, right? So it'll always be incorrect here?


Maybe a future easy issue to clean up?

Atticus29 · 2025-01-22T20:17:17Z

collections/admin/importextended.php

@@ -337,11 +338,14 @@ function importTypeChanged(selectElement){
 									<option value="externalOccurrence"><?= $LANG['EXTERNAL_OCCURRENCE'] ?></option>
 									<option value="observational"><?= $LANG['OBSERVATION'] ?></option>
 								</select>
+								<div class="top-breathing-room-rel danger" style="color: var(--danger-color);">
+									<caption><?= $LANG['ASSOCIATION_UPLOAD_WARNING'] ?></caption>


A little behavioral nudge for the end users who could footgun themselves with bulk uploading associations matching to the correct catalogNumber in an off-target collection (since cross-collection associations are correctly allowed as well).

Atticus29 · 2025-01-22T20:19:11Z

content/lang/collections/admin/importextended.en.php

@@ -33,7 +33,7 @@
 $LANG['FIELD_MAPPING'] = 'Field Mapping';
 $LANG['BATCH_DELETE'] = 'Batch Delete';
 $LANG['NEW_BLANK_RECORD'] = 'Link media to new blank record if catalog number does not exist';
-$LANG['MATCHING_IDENTIFIERS'] = 'Update or delete records with matching identifiers (subject identifier plus objectID or resourceUrl)';
+$LANG['MATCHING_IDENTIFIERS'] = 'Update or delete existing association with matching identifiers (subject identifier plus objectID or resourceUrl)';


Per conversations with @themerekat , minor copy adjustments here.

themerekat · 2025-01-22T22:23:45Z

I think something is going slightly wrong with the exporter's verbatimScientificName (not sure if this is upon export or during the creation of the association). The "hostOf" relationship should be with Apidae

From the export:

From the public display:

Atticus29 · 2025-01-22T22:43:07Z

I think something is going slightly wrong with the exporter's verbatimScientificName (not sure if this is upon export or during the creation of the association). The "hostOf" relationship should be with Apidae

From the export:

From the public display:

Oh interesting. This is an observational association. Maybe we make that a problem for FutureMark?

* swap the occid and occidAssociate in getAssociatedRecords in classes/AssociationManager.php in order to get any results in the external observation export * add third logic block to getAssociatedRecords targetting observations and external observations * begin to remove cruft on 1924-pt2-external-and-observational-occurrences branch * remove more cruft * remove more cruft * respond to code review feedback

Atticus29 added 8 commits December 2, 2024 18:17

create DwcArchiverAssociation.php and implement mapping in setFieldAr…

1f00e00

…r method

finish implementing first version of classes/DwcArchiverAssociation.php

b12ac39

silence association sql print

fa08ff9

add translations, include Resource Relationship extension checkbox, a…

31843c8

…nd begin to implement some of the export logic

fix minor bug

1204989

make association download work. Populate? No.

bc9520e

allow writeOutRecordBlock to handle the union required for two-way as…

a356238

…sociation search

add symbiota native columns to the export

4320a0a

Atticus29 added 3 commits December 9, 2024 16:15

make it so there is no data cutoff for xdebugger

0976003

fix bug where download was not working when association search was be…

8e8464f

…ing used

comment out associatedOccurrences

50604f5

Atticus29 commented Dec 10, 2024

View reviewed changes

Atticus29 added 2 commits December 9, 2024 16:43

remove appendSpecimenDuplicateAssociations, getAssociationJSON, and g…

5db000b

…etAssociationStr

remove some cruft and change some translations

20a3b08

Atticus29 and others added 5 commits December 10, 2024 14:03

fix bug where occurrence download was not respecting association cons…

439e119

…traints

fix bug where ->associationManager->getAssociatedRecords was being ru…

ba6a9c4

…n twice

begin to implement the changes where omoccurassociations is joined to…

229be08

… omoccurrences so that occurrenceID can be queried

association export mod

33ed860

- Load associations via 2 separate queries: -- one using occid -- the other using occidAssociate with the relationship translated to its inverse

Merge pull request #1961 from BioKIC/associations-export-egbot

44fc6f8

association export mod

Atticus29 mentioned this pull request Jan 6, 2025

[Feature request] Export associations as Resource Relationship file #1924

Open

Atticus29 added 8 commits January 15, 2025 11:04

cherry pick copy changes and resolve merge conflicts

77948ac

resolve merge conflicts with Development

5015abf

add warning to users about uniqueness of occurrenceId over other iden…

af00f35

…tifiers

fix typo in recordID

894fa64

add temporary columns to aid in troubleshooting

572d440

fix typos and minor bugs in collections/search/js/searchform.js

5bf4767

resolve merge conflicts with Development

e8e2138

fix bug where was not targeting the correct keys for DWC-formatted do…

61db6fa

…wnload

Atticus29 added 10 commits January 16, 2025 18:21

rename DwcArchiverAssociation to DwcArchiverResourceRelationship

9bcbfc6

remove renamed class file

b50ad33

comment out appendSpecimenDuplicateAssociations call per ticket issue…

3b44f25

…s recommendation

remove no-longer-needed schema change

e3fdff9

remove unnecessary TODO

d7c609f

get more things from the omoccurrences table. Fix bug in setSqlBase l…

5451d3c

…ogic

add left join of the object omoccurrences entry to fetch relatedResou…

8b2f361

…rceID, which required some logic change since omoccurrences does not have objectID nor instanceID

refine relatedResourceID and resourceRelationshipID

8c5128d

make relatedResourceID refer to the object

70076ea

continue improvements to resourceRelationshipID and relatedResourceID

de7712a

Atticus29 commented Jan 22, 2025

View reviewed changes

add missing translations

f266105

Atticus29 marked this pull request as ready for review January 22, 2025 20:31

Atticus29 requested review from egbot and themerekat January 22, 2025 20:31

Atticus29 changed the title ~~1924 export associations as resource relationship~~ [3.3] 1924 export associations as resource relationship Jan 22, 2025

Atticus29 added 2 commits January 28, 2025 10:14

resolve merge conflicts

2f8bb6d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[3.3] 1924 export associations as resource relationship #1933

[3.3] 1924 export associations as resource relationship #1933

Atticus29 commented Dec 4, 2024 •

edited

Loading

themerekat commented Dec 4, 2024

Atticus29 Dec 10, 2024

Atticus29 commented Dec 10, 2024

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

Atticus29 Jan 22, 2025

themerekat commented Jan 22, 2025

Atticus29 commented Jan 22, 2025

[3.3] 1924 export associations as resource relationship #1933

Are you sure you want to change the base?

[3.3] 1924 export associations as resource relationship #1933

Conversation

Atticus29 commented Dec 4, 2024 • edited Loading

Description

QA Notes

Notes/Known Issues

Pull Request Checklist:

Pre-Approval

Post-Approval

themerekat commented Dec 4, 2024

Choose a reason for hiding this comment

Atticus29 commented Dec 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

themerekat commented Jan 22, 2025

Atticus29 commented Jan 22, 2025

Atticus29 commented Dec 4, 2024 •

edited

Loading