Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstash Globus functionality has become overly cumbersome #339

Open
3 tasks done
golaz opened this issue Jul 30, 2024 · 1 comment · May be fixed by #349
Open
3 tasks done

zstash Globus functionality has become overly cumbersome #339

golaz opened this issue Jul 30, 2024 · 1 comment · May be fixed by #349

Comments

@golaz
Copy link
Collaborator

golaz commented Jul 30, 2024

Request criteria

  • I searched the zstash GitHub Discussions to find a similar question and didn't find it.
  • I searched the zstash documentation.
  • This issue does not match the other templates (i.e., it is not a bug report, documentation request, feature request, or a question.)

Issue description

Using zstash with Globus is now much more cumbersome than it used to be; to the point that it may become questionable whether integrating Globus actually adds value to zstash.

Here is the workflow I have to use:

(1) Login to Globus web interface and activate end points

(2) Delete existing globus cfg file

rm ~/.globus-native-apps.cfg

(3) Start interactive zstash test transfer, something like

zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/tmp tmp

During the the interactive test run, zstash will ask to copy and paste authorization code twice. zstash runs but files are not transferred.

(4) Start a second interactive zstash transfer

zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/tmp tmp

This one should complete without issue or any prompt.

(5) Start long transfer (in a screen session to keep it running)

zstash create -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} . 2>&1 | tee zstash/zstash_create_${stamp}.log

Note that this transfer is limited to 48 hours due to Globus token expiration. Given the relatively slow transfer speeds between chrysalis and NERSC HPSS (~100 MB/s), 48 hours is often insufficient for a large simulation.

(6) Using Globus web interface, manually transfer zstash files that were not transferred due to token expiration.

(7) Repeat steps (2) to (4) above. Restart zstash archiving that stopped:

zstash update -v --non-blocking --hpss=globus://nersc/home/g/golaz/E3SMv3.LR/${EXP} 2>&1 | tee zstash/zstash_create_${stamp}.log

I should note that in the not so distant past, only steps (1) and (5) were needed. It is also possible that I'm doing something wrong, in which case I'd like to know what.

@golaz
Copy link
Collaborator Author

golaz commented Jul 30, 2024

Related to #338

@forsyth2 forsyth2 linked a pull request Sep 4, 2024 that will close this issue
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant