feat: Add matching sorts (#497)

* test: Separate match test cases into a testing file

* test: Add match offers benchmark

* test: Add benchmark GH action

* docs: Add benchmarking docs

* feat: Add GetResourceOffers query oldest first

* feat: Add GetJobOffers query oldest first

* fix: Return match result values instead of refs

* test: Lift test store helpers to solver package

* feat: Matcher retrieve offers oldest first

* feat: Add isCheaperOrOlder matching helper

* test: Add TestIsCheaperOrOlder unit test

* feat: Add pricing and age sorting preferences

* test: Add matching and sorting integration test

We integrate at the solver to test matcher and store functionality
together.

* feat: Add service logger to matcher

* chore: Add lookup and match decision add error logs

* refactor: Move adding match decisions into a helper function

* chore: Log matched resource offer decisions count

.github/workflows/bench.yml

@@ -0,0 +1,58 @@
+name: Benchmarks
+
+on:
+  pull_request:
+    branches:
+      - main
+  push:
+    branches:
+      - main
+
+permissions:
+  deployments: write
+  contents: write
+
+jobs:
+  run-solver-benchmarks:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Install golang
+        uses: actions/setup-go@v5
+
+      - name: Run benchmarks
+        run: ./stack benchmarks-solver | tee assets/benchmark-output.txt
+
+      - name: Download previous benchmark data
+        uses: actions/cache@v4
+        with:
+          path: ./cache
+          key: ${{ runner.os }}-benchmark
+
+      - name: Report benchmark results
+        if: github.event_name == 'pull_request'
+        uses: benchmark-action/github-action-benchmark@v1
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          tool: 'go'
+          output-file-path: assets/benchmark-output.txt
+          external-data-json-path: ./cache/benchmark-data.json
+          # Comment when performance degrades >2x
+          comment-on-alert: true
+          # Add summary to action run
+          summary-always: true
+          auto-push: false
+
+      - name: Publish benchmark results
+        if: github.event_name == 'push' && github.ref_name == 'main'
+        uses: benchmark-action/github-action-benchmark@v1
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          tool: 'go'
+          output-file-path: assets/benchmark-output.txt
+          # Add summary to action run
+          summary-always: true
+          # Publish gh-pages chart
+          auto-push: true

LOCAL_DEVELOPMENT.md

@@ -97,6 +97,21 @@ Run the Go unit tests with `./stack unit-tests` and the Hardhat unit tests with
 
 Run the integration tests with `./stack integration-tests`. The integration tests expect all parts of the stack are running, except the request to run a job. See [Using Docker Compose](./LOCAL_DEVELOPMENT.md#using-docker-compose) to run the stack.
 
+#### Benchmarks
+
+We perform relative benchmarking on our solver matching algorithm to avoid performance degradation. The results over time are displayed in charts here: https://lilypad-tech.github.io/lilypad/dev/bench/.
+
+Each CI benchmark test run displays results in the GitHub action summary for the previous and current commit. An [example run](https://github.com/Lilypad-Tech/lilypad/actions/runs/12942901815) shows four measurements that are also reflected in the charts:
+
+- BenchmarkMatchOffers: A summary of overall measurements
+- BenchmarkMatchOffers - ns/op: Nanoseconds per operation
+- BenchmarkMatchOffers - B/op: Number of bytes allocated per operation
+- BenchmarkMatchOffers - allocs/op: Number of allocations per operation
+
+Each op is a run of the matching algorithm over a set of job and resource offers designed to exercise the possible matching decisions.
+
+When the benchmark detects a 2x or greater performance degradation, it displays a commit comment with a perforance alert. See this [example commit](https://github.com/Lilypad-Tech/lilypad/commit/249cca9fe568ecfc6f04813b48ad46ccc0c76258) that intentionally introduces a performance degradation.
+
 ## Notes on tooling
 
 Things should work right out-of-the-box, no extra configuration should be needed as Doppler provides the environment variables that are required with the current setup.

README.md

@@ -1,4 +1,4 @@
-LIKE THIS PROJECT? 
+LIKE THIS PROJECT?
 
 PLEASE STAR US AND HELP US GROW! <3
 
@@ -18,7 +18,7 @@ lilypad run cowsay:v0.0.4 -i Message="moo"
 
 [![Watch the video](https://img.youtube.com/vi/Ep9ML9h8DTE/0.jpg)](https://www.youtube.com/watch?v=Ep9ML9h8DTE)
 
-The current list of modules can be found in the following repositories: 
+The current list of modules can be found in the following repositories:
 
 
 * [lilysay](https://github.com/Lilypad-Tech/lilypad-module-lilysay)
@@ -31,9 +31,13 @@ Containerised job modules can be built and added to the available module list; f
 
 ## Getting started running a Node on Lilypad Network
 
-As a distributed network Lilypad also brings with it the ability to run as a node and contribute to the GPU and compute capabilities. See the documentation on [running a node](https://lilypad.team/node) which contains more details instructions and overview for getting set up. 
+As a distributed network Lilypad also brings with it the ability to run as a node and contribute to the GPU and compute capabilities. See the documentation on [running a node](https://lilypad.team/node) which contains more details instructions and overview for getting set up.
 
-## The Lilypad Community 
+## Performance
+
+We benchmark our solver job matching algorithm: https://lilypad-tech.github.io/lilypad/dev/bench/. See our [benchmark guide](./LOCAL_DEVELOPMENT.md#benchmarks) for more details.
+
+## The Lilypad Community
 
 [Read our Blog](https://lilypad.team/blog)
 
@@ -42,4 +46,3 @@ As a distributed network Lilypad also brings with it the ability to run as a nod
 [Follow us on Twitter/X](https://lilypad.team/x)
 
 [Check out our videos on YouTube](https://lilypad.team/youtube)
-

pkg/solver/controller.go

@@ -283,7 +283,7 @@ func (controller *SolverController) solve(ctx context.Context) error {
 	defer span.End()
 
 	// find out which deals we can make from matching the offers
-	deals, err := matcher.GetMatchingDeals(ctx, controller.store, controller.updateJobOfferState, controller.tracer, controller.meter)
+	deals, err := matcher.GetMatchingDeals(ctx, controller.store, controller.updateJobOfferState, controller.log, controller.tracer, controller.meter)
 	if err != nil {
 		span.SetStatus(codes.Error, "get matching deals failed")
 		span.RecordError(err)

pkg/solver/matcher/match.go

@@ -211,6 +211,7 @@ func (result priceMismatch) attributes() []attribute.KeyValue {
 	}
 }
 
+// TODO(bgins) Rename to validator
 type mediatorMismatch struct {
 	resourceOffer data.ResourceOffer
 	jobOffer      data.JobOffer
@@ -253,19 +254,19 @@ func matchOffers(
 	jobOffer data.JobOffer,
 ) matchResult {
 	if resourceOffer.Spec.CPU < jobOffer.Spec.CPU {
-		return &cpuMismatch{
+		return cpuMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
 	}
 	if resourceOffer.Spec.GPU < jobOffer.Spec.GPU {
-		return &gpuMismatch{
+		return gpuMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
 	}
 	if resourceOffer.Spec.RAM < jobOffer.Spec.RAM {
-		return &ramMismatch{
+		return ramMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
@@ -275,7 +276,7 @@ func matchOffers(
 	if len(jobOffer.Spec.GPUs) > 0 {
 		// Mismatch if job offer requests VRAM but resource provider has none
 		if len(resourceOffer.Spec.GPUs) == 0 {
-			return &vramMismatch{
+			return vramMismatch{
 				jobOffer:      jobOffer,
 				resourceOffer: resourceOffer,
 			}
@@ -285,23 +286,23 @@ func matchOffers(
 		largestResourceOfferVRAM := getLargestVRAM(resourceOffer.Spec.GPUs)
 		largestJobOfferVRAM := getLargestVRAM(jobOffer.Spec.GPUs)
 		if largestResourceOfferVRAM < largestJobOfferVRAM {
-			return &vramMismatch{
+			return vramMismatch{
 				jobOffer:      jobOffer,
 				resourceOffer: resourceOffer,
 			}
 		}
 	}
 
 	if resourceOffer.Spec.Disk < jobOffer.Spec.Disk {
-		return &diskSpaceMismatch{
+		return diskSpaceMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
 	}
 
 	moduleID, err := data.GetModuleID(jobOffer.Module)
 	if err != nil {
-		return &moduleIDError{
+		return moduleIDError{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 			err:           err,
@@ -320,7 +321,7 @@ func matchOffers(
 		}
 
 		if !hasModule {
-			return &moduleMismatch{
+			return moduleMismatch{
 				jobOffer:      jobOffer,
 				resourceOffer: resourceOffer,
 				moduleID:      moduleID,
@@ -330,38 +331,39 @@ func matchOffers(
 
 	// we don't currently support market priced resource offers
 	if resourceOffer.Mode == data.MarketPrice {
-		return &marketPriceUnavailable{
+		return marketPriceUnavailable{
 			resourceOffer: resourceOffer,
 		}
 	}
 
 	// if both are fixed price then we filter out "cannot afford"
 	if resourceOffer.Mode == data.FixedPrice && jobOffer.Mode == data.FixedPrice {
 		if resourceOffer.DefaultPricing.InstructionPrice > jobOffer.Pricing.InstructionPrice {
-			return &priceMismatch{
+			return priceMismatch{
 				jobOffer:      jobOffer,
 				resourceOffer: resourceOffer,
 				moduleID:      moduleID,
 			}
 		}
 	}
 
+	// TODO(bgins) Rename to validator
 	mutualMediators := data.GetMutualServices(resourceOffer.Services.Mediator, jobOffer.Services.Mediator)
 	if len(mutualMediators) == 0 {
-		return &mediatorMismatch{
+		return mediatorMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
 	}
 
 	if resourceOffer.Services.Solver != jobOffer.Services.Solver {
-		return &solverMismatch{
+		return solverMismatch{
 			jobOffer:      jobOffer,
 			resourceOffer: resourceOffer,
 		}
 	}
 
-	return &offersMatched{
+	return offersMatched{
 		jobOffer:      jobOffer,
 		resourceOffer: resourceOffer,
 	}