Give bench_local the option to run stable benchmarks

[Nadrieril]

In #1777 I noticed that I couldn't run category: stable benchmarks locally; this PR makes this possible.

We can now specify primary/secondary/stable in --include and --exclude. If there's no --include/--exclude it defaults to --exclude stable which preserves today's behavior. This does change behavior for anyone running bench_local --exclude ...: they would need to add --exclude ...,stable to bench the same set as today.

[Kobzol]

I think that unless @nnethercote objects, we could just allow stable benchmarks by default, by removing the line that filters them out.

But for what it's worth, I don't think that it is intuitive to deal with categories in --include/--exclude, since these flags are about benchmark names, not categories. I would rather add a new flag, --category/--categories, which would have primary,secondary as the default value.

[Nadrieril]

Hm, a different flag does not make sense to me, I could want --include primary,encoding

[Kobzol]

--include filters benchmarks by name, not category, so --include primary would have to mean "include the whole primary category". But that would also mean that there cannot be any overlap between benchmark names and categories, and is, well, a horrible hack :)

[Nadrieril]

Yeah that's how I thought about it. Doesn't feel more a hack than lint groups and lints using an identical namespace like #[allow(unused)]/#[allow(dead_code)] 🤷‍♀️ I'm fine with anything that lets me run stable benches tbh

[nnethercote]

Some thoughts:

• I like that stable benchmarks aren't run by default.
  • The style-servo benchmark was particularly large, and when it was part of the normal suite I almost always excluded it when running locally.
  • There is some overlap between the stable benchmarks and the primary, e.g. two different versions of regex, html5ever, piston-image, syn.
  • I think the stable benchmarks are only really useful for the long-term performance picture, and I would almost never run them locally myself.
• Having said that, I agree it's reasonable to be want to run them locally.
• My initial feeling is that a separate flag --category would make more sense than this PR's approach. It would default to Primary,Secondary. --include/--exclude would then be applied within the chosen categories.

Hm, a different flag does not make sense to me, I could want --include primary,encoding

Would that run all the primary benchmarks + encoding from the stable benchmarks? Hmm, is that likely to happen in practice?

We also have a problem that some of the stable benchmark names are prefixes of some of the primary benchmark names, e.g. the ones I mentioned above, so it would be impossible to run the primary benchmark syn-1.0.89 without also running the stable benchmark syn.

Also, if it's done with just --include/--exclude how are conflicting category/individual specifications combined? E.g. --exclude stable --include syn?

[Kobzol]

I agree with @nnethercote, and I would suggest #1780 as an alternative solution to your problem.

[Nadrieril]

Hmm, is that likely to happen in practice?

Probably not tbf. I always use an explicit list myself.

#1780 works for me!