#14406: Add perf test for reduce scatter

[Aswinmcw]

Ticket

#14406

What's changed

Adds perf test for reduce scatter T3k ring and line, N300 ring and line

Checklist

• Post commit CI passes
• Blackhole Post commit (if applicable)
• Model regression CI testing passes (if applicable)
• Device performance regression CI testing passes (if applicable)
• New/Existing tests provide coverage for changes

[SeanNijjar]

Looks good overall but there are two things that look off here (potentially). I'm not sure which dataformat being used for each test (can you please add that), and I think the Op BW equation is bugged. I'll revisit that today and get back to you with the right equation because op BW should always be >= link BW and right now it is much lower.

[SeanNijjar]

I double checked the equation in the issue for ring reduce scatter op bandwidth, it was incorrect. I corrected it to input_tensor_volume / longest_device_fw_time for the "per chip" op bandwidth. Total op BW is a little squishy for the full cluster. That may eventually end up being a more useful measurement but this is useful to track right now (perhaps a later iteration of this work can also express cluster-level op BW

[SeanNijjar]

Approving now pending updates to BW calculations:

for ring reduce scatter it should match line reduce scatter: input_tensor_volume / longest_device_fw_time

Also I realized the reduce scatter eth BW is incorrect. Since we send/receive the full input tensor volume through each chip's ethernet, we should just do input_tensor_volume \ / longest_erisc_fw_time