Not reporting on all Spark clusters

[sdewitt-newrelic]

When listing all clusters at the start of the spark collection, we use ListAll instead of List, which does not allow for pagination. So we are missing clusters. Need to use List with the listing.Iterator instead.

When we do this, it will significantly increase the time of PollMetrics. But we have to pull the list each time in case clusters have changed state or have been terminated, or new ones started. What should we do to account for this? Can we use a filter when we call List?

We could also add the option to specify a list of cluster IDs instead of automatic discovery.