Activities and Priorities

Current state

FTS3 multi-dimentional scheduling offers two features:

• activities shares, which allow to divide the assigned slots acording to weights decided by the VO
• priorities, which allow to reshuffle the jobs within an activity share

CMS does not rely on these features, PhEDEx has its own intricate scheduling mechanism, and FTS3 acts simply as a FIFO queue. Rucio provides interface for both features; however, ATLAS is only using activity shares, as setting different priorities may cause starvation.

Proposal:

We propose to introduce the following activity shares in CMS Rucio (initial weights to be decided):

• tier-0 transfers
• pre-production transfers
• post-production transfers
• debug transfers
• popularity based replication
• user transfers
• crab staging
• ASO
• default (pre-existing)

The activity is specified as an attribute of the replication rule. The weights are defined in FTS configuration and can be adjusted via API using VO production role.

References:

1. FTS scheduling: http://fts3-docs.web.cern.ch/fts3-docs/docs/features.html#multidimensional-scheduler
2. PhEDEx scheduling: https://twiki.cern.ch/twiki/bin/view/CMS/PhedexAdminDocsPriorityQueues
3. Brief summary of prioritization in PhEDEx:

• Priority levels (high, normal, low, reserved) are specified by user in transfer request.
• Approved request turn into subscriptions.
• Subscribed files are allocated for transfer in order of priority and the time the data was requested, older request first.
• Files are allocated for routing until 50 TB request windows per priority are filled.
• Priority changes are propagated to the allocated file requests.
• Once data are routed, transfer tasks are created with task_priority, time_assigned, and rank . The rank is ordered by priority and then the file's logical name.
• Transfer tasks are fetched by the site FileDownload agent, sorted by time_assigned and rank and submitted to FTS in that order.