Add support for intra.broker.goals in anomaly detection / self healing

[ecojan]

This PR resolves linkedin#1264 by enabling the self-healing for intra-broker goals. This however does not account for race conditions between Cluster level Goals and will only trigger after said goals are finished.
The scope of this PR is to allow self-healing on IntraBroker goals for now (as future work might make this obsolete as intra-broker goals and regular goals might be made to work together).

New configuration:

anomaly.detection.intra.broker.goals= List of intra-broker goals for the Anomaly detector to look for
self.healing.intra.broker.goals= List of Intra broker Goals for self-healing
self.healing.intra.broker.goal.violation.enabled= If the self-healing should be enabled for intra-broker-goals

New Anomaly Type: INTRA_BROKER_GOAL_VIOLATION

A new type of Anomaly that needs to be treated differently from the regular GOAL_VIOLATIONS.
When configuring for the rebalance parameters for this violation, we are skipping hard goals check and ignoring proposalsCache.
We are skipping hard goals check because intra-broker goals may not be hard goals (at the current given time).
We are ignoring proposalsCache because the proposals model takes into account all given default goals. If we add the intraBrokersGoals this will not work properly as currently, disk-granularity goals do not work with broker-granularity goals.

New Anomaly detectors: IntraBrokerGoalViolationDetector

This anomaly detector is only looking for the anomaly.detection.intra.broker.goal and unlike the GoalViolationDetector, it's using a cluster model that is generating replica placement on disks as well.
This anomaly detector mainly functions for INTRA_BROKER_GOAL_VIOLATIONS.

What is missing from this feature

• missing representation on INTRA_BROKER_GOAL_VIOLATIONS in CruiseControl UI when an anomaly is detected (as this is a structure in cruise-control-ui project)

[amuraru]

@ecojan one particular aspect which is not yet clear to me is the order of the INTRA and INTER broker goal violations.
is the new intra-broker-goal-violation detector/healer run before the intra one?

[ecojan]

@ecojan one particular aspect which is not yet clear to me is the order of the INTRA and INTER broker goal violations. is the new intra-broker-goal-violation detector/healer run before the intra one?

My expectancy is GOAL_VIOLATIONS followed by INTRA_BROKER_GOAL_VIOLATIONS.
As stated on

cruise-control/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/detector/AnomalyDetectorManager.java

Line 243 in 908ae8f

scheduleDetectorAtFixedRate(INTRA_BROKER_GOAL_VIOLATION, _intraBrokerGoalViolationDetector);

the detectors are scheduled on a fixed rate and in order (have not checked the inner workings for maintaining order on ScheduledExecutorService from the java.util.concurrent package).
For INTRA broker goal violation detector, we reuse the same anomalyDetection interval from the INTER broker goal violation detector.

[amuraru]

thanks for the pointer - yeah - they all seem to be scheduled concurrently with the same (default) interval.
This means that whichever detector finishes first will be the one scheduling potential fix tasks.
My idea was if possible to consider running INTRA_BROKER_GOAL_VIOLATIONS more frequently and thus giving it a chance to run before the GOAL_VIOLATIONS one - the reason being that it can level the usage within a broker before other paritions are migrated to it as part of the global goal_violation detector.
but this is not mandatory at this stage.

[ecojan]

did a rebase, waiting for build status

[dobrerazvan]

🚢

[ilievladiulian]

Is there a reason why the cluster model if first set to null and afterwards with an actual value?