Support constrained optimization for timeline plot #678

not522 · 2023-11-02T07:40:02Z

Contributor License Agreement

This repository (optuna-dashboard) and Goptuna share common code.
This pull request may therefore be ported to Goptuna.
Make sure that you understand the consequences concerning licenses and check the box below if you accept the term before creating this pull request.

I agree this patch may be ported to Goptuna by other Goptuna contributors.

Reference Issues/PRs

This feature corresponds to optuna/optuna#4877.

What does this implement/fix? Explain your changes.

This PR supports constrained optimization for timeline plot. Same as Optuna, it applies only to COMPLETE trials.

codecov · 2023-11-02T07:44:45Z

Codecov Report

Merging #678 (3628c31) into main (0e1feb3) will not change coverage.
Report is 22 commits behind head on main.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #678   +/-   ##
=======================================
  Coverage   62.88%   62.88%           
=======================================
  Files          35       35           
  Lines        2250     2250           
=======================================
  Hits         1415     1415           
  Misses        835      835

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

not522 · 2023-11-02T08:37:14Z

@HideakiImamura @nabenabe0928 Could you review this PR?

nabenabe0928

Thank you for the PR!
I left some comments")

nabenabe0928 · 2023-11-02T09:58:03Z

optuna_dashboard/ts/components/GraphTimeline.tsx

+    if (s == "Complete") {
+      const feasibleTrials = bars.filter((t) =>
+        t.constraints.every((c) => c <= 0)
+      )
+      const infeasibleTrials = bars.filter((t) =>
+        t.constraints.some((c) => c > 0)
+      )
+      if (feasibleTrials.length > 0) {
+        traces.push(makeTrace(feasibleTrials, "Complete", cm[s]))
+      }
+      if (infeasibleTrials.length > 0) {
+        traces.push(makeTrace(infeasibleTrials, "Infeasible", "#cccccc"))
+      }
+    } else {
+      traces.push(makeTrace(bars, s, cm[s]))
+    }


What about early return?

Suggested change

if (s == "Complete") {

const feasibleTrials = bars.filter((t) =>

t.constraints.every((c) => c <= 0)

)

const infeasibleTrials = bars.filter((t) =>

t.constraints.some((c) => c > 0)

)

if (feasibleTrials.length > 0) {

traces.push(makeTrace(feasibleTrials, "Complete", cm[s]))

}

if (infeasibleTrials.length > 0) {

traces.push(makeTrace(infeasibleTrials, "Infeasible", "#cccccc"))

}

} else {

traces.push(makeTrace(bars, s, cm[s]))

}

if (s !== "Complete") {

traces.push(makeTrace(bars, s, cm[s]))

continue

}

const feasibleTrials = bars.filter((t) =>

t.constraints.every((c) => c <= 0)

)

const infeasibleTrials = bars.filter((t) =>

t.constraints.some((c) => c > 0)

)

if (feasibleTrials.length > 0) {

traces.push(makeTrace(feasibleTrials, "Complete", cm[s]))

}

if (infeasibleTrials.length > 0) {

traces.push(makeTrace(infeasibleTrials, "Infeasible", "#cccccc"))

}

I don't think early return is appropriate here since the codes inside the brackets are at the same level. For example, if we add another if statement for "Waiting", it should be like this.

if (s === "Complete") { ... } else if (s === "Waiting") { ... } else { ... }

not like this.

if (s !== "Complete" && s !== "Waiting") { ... continue } if (s === "Complete") { ... continue } ...

nabenabe0928 · 2023-11-02T10:01:06Z

optuna_dashboard/ts/components/GraphTimeline.tsx

-    if (bars.length === 0) {
-      continue
-    }
+  const makeTrace = (bars: Trial[], name: string, color: string) => {


It is a very minor comment, but what about replacing name with state?

Suggested change

const makeTrace = (bars: Trial[], name: string, color: string) => {

const makeTrace = (bars: Trial[], state: string, color: string) => {

HideakiImamura

Thanks for the PR. I have a minor comment. PTAL.

optuna_dashboard/ts/components/GraphTimeline.tsx

HideakiImamura · 2023-11-02T11:28:11Z

I confirmed that the change of this PR generate the following plot for the following script.

    # Study with trials having certain duration and constriants
    def objective_with_duration(trial: optuna.Trial) -> float:
        trial.set_user_attr("constraint", [trial.number % 2])
    
        time.sleep(0.01)
        if trial.number % 3 == 0:
            raise optuna.TrialPruned()
        if trial.number % 7 == 0:
            return float("nan")
        return 0.0

    def constraints(trial: optuna.Trial) -> list[float]:
        return trial.user_attrs["constraint"]

    study = optuna.create_study(
        study_name="trials-with-duration", 
        storage=storage,
        sampler=optuna.samplers.NSGAIISampler(constraints_func=constraints),
    )

    trial = study.ask({"x": FloatDistribution(0, 10), "y": CategoricalDistribution(["Foo", "Bar"])})  # To create a running trial
    trial.set_user_attr("constraint", [1])
    time.sleep(0.01)

    study.optimize(objective_with_duration, n_trials=10)

    study.enqueue_trial({"x": 0, "y": 10})  # To create a waiting trial
    time.sleep(0.01)

HideakiImamura · 2023-11-02T11:32:30Z

I noticed that the running and waiting trials are not appropriately plotted. Also, it seems that the waiting trials are not plotted properly even in Optuna. As neither of these issues are due to the changes in this PR, I believe it would be best to address these in a separate PR.

Optuna Dashboard	Optuna

Co-authored-by: Hideaki Imamura <[email protected]>

not522 · 2023-11-08T07:03:09Z

Thank you for your reviews. I have updated this PR responding to your comments. PTAL.

HideakiImamura

Thanks for the update. LGTM.

nabenabe0928

Thanks for the update! LGTM

Support constrained optimization for timeline plot

6bec7e3

not522 assigned HideakiImamura and nabenabe0928 Nov 2, 2023

nabenabe0928 reviewed Nov 2, 2023

View reviewed changes

HideakiImamura reviewed Nov 2, 2023

View reviewed changes

optuna_dashboard/ts/components/GraphTimeline.tsx Outdated Show resolved Hide resolved

not522 and others added 4 commits November 8, 2023 15:29

Update optuna_dashboard/ts/components/GraphTimeline.tsx

e0bc525

Co-authored-by: Hideaki Imamura <[email protected]>

Remove excessive parenthesis

ff6ce9d

Use strict equality operator

cbbe56a

Change argument name to state

3628c31

HideakiImamura approved these changes Nov 9, 2023

View reviewed changes

nabenabe0928 approved these changes Nov 10, 2023

View reviewed changes

nabenabe0928 removed their assignment Nov 10, 2023

keisuke-umezawa approved these changes Nov 12, 2023

View reviewed changes

keisuke-umezawa merged commit 285e5e0 into optuna:main Nov 12, 2023
10 checks passed

not522 deleted the timeline-constraint branch November 13, 2023 03:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support constrained optimization for timeline plot #678

Support constrained optimization for timeline plot #678

not522 commented Nov 2, 2023

codecov bot commented Nov 2, 2023 •

edited

Loading

not522 commented Nov 2, 2023

nabenabe0928 left a comment

nabenabe0928 Nov 2, 2023

not522 Nov 8, 2023

nabenabe0928 Nov 2, 2023

HideakiImamura left a comment

HideakiImamura commented Nov 2, 2023

HideakiImamura commented Nov 2, 2023

not522 commented Nov 8, 2023

HideakiImamura left a comment

nabenabe0928 left a comment

	const makeTrace = (bars: Trial[], name: string, color: string) => {
	const makeTrace = (bars: Trial[], state: string, color: string) => {

Support constrained optimization for timeline plot #678

Support constrained optimization for timeline plot #678

Conversation

not522 commented Nov 2, 2023

Contributor License Agreement

Reference Issues/PRs

What does this implement/fix? Explain your changes.

codecov bot commented Nov 2, 2023 • edited Loading

Codecov Report

not522 commented Nov 2, 2023

nabenabe0928 left a comment

Choose a reason for hiding this comment

nabenabe0928 Nov 2, 2023

Choose a reason for hiding this comment

not522 Nov 8, 2023

Choose a reason for hiding this comment

nabenabe0928 Nov 2, 2023

Choose a reason for hiding this comment

HideakiImamura left a comment

Choose a reason for hiding this comment

HideakiImamura commented Nov 2, 2023

HideakiImamura commented Nov 2, 2023

not522 commented Nov 8, 2023

HideakiImamura left a comment

Choose a reason for hiding this comment

nabenabe0928 left a comment

Choose a reason for hiding this comment

codecov bot commented Nov 2, 2023 •

edited

Loading