diff --git a/README.md b/README.md index 30cc2b2..b792913 100644 --- a/README.md +++ b/README.md @@ -10,104 +10,124 @@ these in code that doesnt need to require all of arnesi. Also arnesi is hard to update. [original arnesi docs](http://common-lisp.net/project/bese/docs/arnesi/html/Reducing_0020and_0020Collecting.html) - + +At this point the API and performance profile has diverged from ARNESI ## API -### make-collector / with-collector / with-mapping-collector +### Definitions - as applied to this library -Create a collector function. +Aggregators: functions / objects which take many values and combine + them in some logical way. All aggregators accept `N` values and, + after aggregating each of them, return the new aggregate value -A Collector function will collect, into a list, all the values -passed to it in the order in which they were passed. If the -callector function is called without arguments it returns the -current list of values. +Reducers: aggregate values by combining new values into a single value, + eg: #'+ can be used to reduce successive numbers into their sum -``` - (with-collector (col) - (col 1) - (col 2) - (col 3) - (col)) => (1 2 3) -``` +Collectors: aggregate items into a list by adding each item to the end + of a list. -Mapping collectors mutate the collected value while collecting it. +Appenders: aggregate items into a list by appending each item to the + end. If single, the item is collected, if a list, it is appended + eg: (app (1) 2 (3 (4))) => (1 2 3 (4)) -``` - (with-mapping-collector (col (x) (* 2 x)) - (col 1) - (col 2) - (col 3) - (col)) => (2 4 6) -``` +Pusher: collect items (by push) at the beginning of a list +### Simple aggregators -### make-reducer / with-reducer +Simple aggregators are lambdas which do not signal (are not +filterable), and store their data in local places. -Create a function which, starting with INITIAL-VALUE, reduces -any other values into a single final value. +#### append-at-end, collect-at-end -FUNCTION will be called with two values: the current value and -the new value, in that order. FUNCTION should return exactly one -value. +Macros for inlining collection at the end of a list by providing +places for each operation -The reducing function can be called with n arguments which will -be applied to FUNCTION one after the other (left to right) and -will return the new value. +#### make-simple-collector / make-simple-appender -If the reducing function is called with no arguments it will -return the current value. +Quickest possible function based implementation of a collector / +appender. Returns (lambda (&rest values) {do} collected-values), +The `-to-place` variants are macros that build the same function, +but with the head of the list stored in a user-provided place -``` -Example: - (setf r (make-reducer #'+ :initial-value 5)) - (funcall r 0) => 5 - (funcall r 1 2) => 8 - (funcall r) => 8 -``` +### Nonsimple aggregators -### make-appender / with-appender / with-mapping-appender +Non simple aggregators are funcallable CLOS instances with a type +heirachy rooted at `value-aggregator`. -Create an appender function. +These type of aggregators support a standard set of operations and +signals. -An Appender will append any arguments into a list, all the values -passed to it in the order in which they were passed. If the -appender function is called without arguments it returns the -current list of values. +#### `operate` / `deoperate` -``` - (with-appender (app) - (app '(1 2)) - (app '(2 3)) - (app '(3 4)) - (app)) => (1 2 2 3 3 4) -``` +These methods perform the correct operation for the type of aggregator +(EG: reducing, appending, collecting, pushing). Deoperate attempts to +undo the operation (currently only defined for the list types). -Mapping appenders mutate the collected values while collecting them. +#### `should-aggregate?` -``` - (with-mapping-appender (app (l) (mapcar #'(lambda (x) (* 2 x)) l)) - (app '(1 2)) - (app '(2 3)) - (app '(3 4)) - (app)) => (2 4 4 6 6 8) -``` +Given an aggregator and a value, should we include the value in our +collection. Used in conjunction with the skip restart to orchestrate +skipping items. + +#### Signals and Restarts + +* collectors-signals:aggregating - signaled when we begin aggregating a value +* collectors-signals:done-aggregating - signaled when we finish aggregating a value +* collectors-signals:skip - *restart* skip aggregating this item (used to filter out nils etc) +* collectors-signals:use-value - *restart* aggregate a different value instead (used for + mapping) +#### make-collector, make-pusher, make-reducer, make-appender -### make-string-builder / with-string-builder / with-string-builder-output +Creates a funcallable instance to perform the expected operation -Create a function that will build up a string for you Each call to the -function with arguments appends those arguments to the string with an -optional delimiter between them. + + +#### Strings: make-formatter (fn), string-formatter (class), with-formatter (macro) + +String formatters accept a format-string and list of arguments, and +use format to process the two into a string. If a stream is provided +we write that string to the stream. We also always concatenate the +results of all formatter calls. The function (as all aggregators) +returns the concatenated results of all calls. + +Optionally a delimiter will be written between each call to the +formatter. + +A provided stream will be written to as each formatter call is made. + +#### Strings: make-string-builder / with-string-builder / with-string-builder-output + +Create a function that will build up a string for you. Each call to +the function concatenates all arguments (coerced to string via princ) +into the result. if ignore-empty-strings-and-nil is true neither empty strings nor nil -will be printed to the stream +will be collect to the stream / aggregate. (Delimiters will also be +elided) + +Optionally a delimiter will be written between each call to the +formatter. -A call to the function with no arguments returns the output string +A provided stream will be written to as each formatter call is made. + +### Context Macros `with-collector` & `with-collector-output` + +Create a lexical function that calls a new aggregator of the requested +type. When using the `-output` variants, the aggregate value is +returned from the form. Otherwise, the value of the last form is +returned per-usual. + +``` + (with-collector (col) + (col 1) ; (1) + (col 2) ; (1 2) + (col 3) ; (1 2 3) + (col)) => (1 2 3) +``` -with-string-builder-output returns the collected string as the value -of the "with" form ## Authors diff --git a/collectors.lisp b/collectors.lisp index 550ff29..896d2ef 100644 --- a/collectors.lisp +++ b/collectors.lisp @@ -1,8 +1,21 @@ ;; -*- lisp -*- +(cl:defpackage :collectors-signals + (:export + ;; signals and restarts + #:aggregating #:skip #:new-value #:value #:aggregator + #:done-aggregating #:aggregate)) + (cl:defpackage :collectors - (:use :cl :cl-user) + (:use :cl :cl-user :collectors-signals) (:export + #:collect-at-end + #:append-at-end + #:make-simple-collector + #:make-simple-appender + #:make-simple-collector-to-place + #:make-simple-appender-to-place + #:with-collector #:with-collector-output #:with-collectors @@ -27,6 +40,108 @@ (in-package :collectors) +(eval-when (:compile-toplevel :load-toplevel :execute) + (define-condition collectors-signals:aggregating () + ((collectors-signals:value + :accessor collectors-signals:value + :initarg :value :initform nil) + (collectors-signals:aggregator + :accessor collectors-signals:aggregator + :initarg :aggregator :initform nil))) + + (define-condition collectors-signals:done-aggregating () + ((collectors-signals:aggregate + :accessor collectors-signals:aggregate + :initarg :aggregate :initform nil) + (collectors-signals:aggregator + :accessor collectors-signals:aggregator + :initarg :aggregator :initform nil))) + + (defmacro with-signal-context ((value after-values aggregator) &body body) + (alexandria:with-unique-names (new-value) + `(with-simple-restart + (collectors-signals:skip "Skip aggregating ~A into ~A" ,value ,aggregator) + (restart-case (signal 'aggregating :value ,value :aggregator ,aggregator) + (collectors-signals:new-value (,new-value) + :report "Aggregate a new value instead" + (setf ,value ,new-value))) + (prog1 (progn ,@body) + (signal 'done-aggregating :after-values ,after-values :aggregator ,aggregator)))))) + +(defmacro collect-at-end (head-place tail-place values-place) + "Macros to ease efficient collection at the end of a list" + (alexandria:with-unique-names (a) + `(dolist (,a ,values-place) + (let ((c (cons ,a nil))) + (when (null ,head-place) (setf ,head-place c)) + (unless (null ,tail-place) (setf (cdr ,tail-place) c)) + (setf ,tail-place c))))) + +(defmacro collect-at-end-with-signals (head-place tail-place values-place + aggregator-place post-values-place) + "Macros to ease efficient collection at the end of a list" + (alexandria:with-unique-names (a) + `(dolist (,a ,values-place) + (with-signal-context (,a ,post-values-place ,aggregator-place) + (let ((c (cons ,a nil))) + (when (null ,head-place) (setf ,head-place c)) + (unless (null ,tail-place) (setf (cdr ,tail-place) c)) + (setf ,tail-place c)))))) + +(defmacro append-at-end (head-place tail-place values-place) + "Macros to ease efficient collection (with appending) at the end of a list" + (alexandria:with-unique-names (a) + `(dolist (,a ,values-place) + (typecase ,a + (list (collect-at-end ,head-place ,tail-place ,a)) + (t (let ((c (cons ,a nil))) + (when (null ,head-place) (setf ,head-place c)) + (unless (null ,tail-place) (setf (cdr ,tail-place) c)) + (setf ,tail-place (last c)))))))) + +(defmacro append-at-end-with-signals (head-place tail-place values-place + aggregator-place post-values-place) + "Macros to ease efficient collection (with appending) at the end of a list" + (alexandria:with-unique-names (a) + `(dolist (,a ,values-place) + (with-signal-context (,a ,post-values-place ,aggregator-place) + (typecase ,a + (list (collect-at-end-with-signals + ,head-place ,tail-place ,a ,aggregator-place ,post-values-place)) + (t (let ((c (cons ,a nil))) + (when (null ,head-place) (setf ,head-place c)) + (unless (null ,tail-place) (setf (cdr ,tail-place) c)) + (setf ,tail-place (last c))))))))) + +(defun make-simple-collector () + "A fastest possible, fewest frills collector suitable to places where efficiency matters" + (let ( head tail ) + (lambda (&rest values) + (collect-at-end head tail values) + head))) + +(defmacro make-simple-collector-to-place (place) + `(let ( tail ) + (lambda (&rest values) + (collect-at-end ,place tail values) + ,place))) + +(defun make-simple-appender () + "A fastest possible, fewest frills collector suitable to places where efficiency matters + that appends any values that re lists" + (let ( head tail ) + (lambda (&rest values) + (append-at-end head tail values) + head))) + +(defmacro make-simple-appender-to-place (place) + "A fastest possible, fewest frills collector suitable to places where efficiency matters + that appends any values that re lists" + `(let ( tail ) + (lambda (&rest values) + (append-at-end ,place tail values) + ,place))) + ;;;; * Reducing and Collecting ;;;; ** Reducing @@ -58,11 +173,22 @@ (defgeneric operate (aggregator values) (:documentation "Perform the aggregation operation on the aggregator for the values") - (:method :after ((o value-aggregator) values - &aux (value (value o))) + (:method :around ((o value-aggregator) values + &aux (places (alexandria:ensure-list (place-setter o)))) (declare (ignore values)) - (dolist (ps (alexandria:ensure-list (place-setter o))) - (funcall ps value)))) + (handler-bind + ((aggregating + (lambda (c) + (when (eql o (aggregator c)) + (unless (should-aggregate? (aggregator c) (value c)) + (invoke-restart 'skip))))) + (done-aggregating + (lambda (c) + (when (eql o (aggregator c)) + (dolist (p places) + (funcall p (value o))))))) + (call-next-method) + (value o)))) (defclass reducer (value-aggregator) ((operation :accessor operation :initarg :operation :initform nil)) @@ -91,12 +217,12 @@ Example: (defmethod operate ((o reducer) values) - (dolist (n (alexandria:ensure-list values)) - (setf (value o) - (if (value o) - (funcall (operation o) (value o) n) - n))) - (value o)) + (dolist (v (alexandria:ensure-list values)) + (with-signal-context (v (value o) o) + (setf (value o) + (if (value o) + (funcall (operation o) (value o) v) + v))))) ;;;; reducing is the act of taking values, two at a time, and ;;;; combining them, with the aid of a reducing function, into a @@ -150,9 +276,8 @@ FUNCTION and INITIAL-VALUE are passed directly to MAKE-REDUCER." (defmethod operate ((o pusher) values) (dolist (v (alexandria:ensure-list values)) - (when (should-aggregate? o v) - (push v (value o)))) - (value o)) + (with-signal-context (v (value o) o) + (push v (value o))))) (defclass collector (list-aggregator) ((tail :accessor tail :initarg :tail :initform nil)) @@ -167,14 +292,7 @@ FUNCTION and INITIAL-VALUE are passed directly to MAKE-REDUCER." (setf (tail o) (last (value o)))) (defmethod operate ((o collector) values) - (dolist (v (alexandria:ensure-list values)) - (when (should-aggregate? o v) - (let ((new-cons (cons v nil))) - (if (value o) - (setf (cdr (tail o)) new-cons) - (setf (value o) new-cons)) - (setf (tail o) new-cons)))) - (value o)) + (collect-at-end-with-signals (value o) (tail o) values o (value o))) (defmethod deoperate ((o list-aggregator) to-remove &key test key @@ -216,7 +334,7 @@ FUNCTION and INITIAL-VALUE are passed directly to MAKE-REDUCER." (:metaclass closer-mop:funcallable-standard-class)) (defmethod operate ((o appender) values) - (call-next-method o (apply #'append (mapcar #'alexandria:ensure-list values)))) + (append-at-end-with-signals (value o) (tail o) values o (values o))) (defun make-appender (&key initial-value place-setter) (make-instance 'appender :initial-value initial-value :place-setter place-setter)) @@ -320,8 +438,7 @@ binds *print-pretty* to nil (setf (has-written? o) t) (setf (value o) (concatenate 'string (value o) new-part)) (when (output-stream o) - (write-string new-part (output-stream o))))) - (value o)) + (write-string new-part (output-stream o)))))) (defun make-formatter (&key delimiter stream pretty) "Create a string formatter collector function. @@ -376,7 +493,7 @@ This form returns the result of that formatter" (*print-pretty* (pretty? o))) (setf values (alexandria:ensure-list values)) (dolist (v values) - (when (should-aggregate? o v) + (with-signal-context (v (value o) o) (setf v (typecase v (string v) (t (princ-to-string v)))) @@ -392,8 +509,7 @@ This form returns the result of that formatter" (t (when out (write-string v out)) (setf (value o) v))) - (setf (has-written? o) t))) - (value o)) + (setf (has-written? o) t)))) (defun make-string-builder (&key delimiter ignore-empty-strings-and-nil stream) (make-instance 'string-builder @@ -428,6 +544,20 @@ This form returns the result of that formatter" ,@body (,name))) +(defun mapping-aggregation-context (body-fn &key aggregator map-fn) + (handler-bind + ((aggregating + (lambda (c) + (when (eql aggregator (aggregator c)) + (invoke-restart 'new-value (funcall map-fn (value c))))))) + (funcall body-fn))) + +(defmacro map-aggregation ((aggregator fn-spec) &body body) + `(mapping-aggregation-context + (lambda () ,@body) + :aggregator ,aggregator + :map-fn ,fn-spec)) + ;;;; Mapping collectors (defmacro with-mapping-collector ((name fn-args &body fn-body) &body body) @@ -445,21 +575,16 @@ This form returns the result of that formatter" " (alexandria:with-unique-names (col flet-args) `(let ((,col (make-collector))) - (flet ((,name (&rest ,flet-args) - (if ,flet-args - (funcall ,col (apply (lambda ,fn-args ,@fn-body) - ,flet-args)) - (funcall ,col)))) - ,@body)))) + (map-aggregation (,col (lambda ,fn-args ,@fn-body)) + (flet ((,name (&rest ,flet-args) (apply ,col ,flet-args))) + ,@body))))) (defmacro with-mapping-appender ((name fn-args &body fn-body) &body body) "Like a with-appender, but instead of a name we take a function spec - if you call the resultant function with no arguments, you get the - collection so far - if you call it with arguments the results of calling your function spec are - collected + calling the function will appen + (with-mapping-appender (app (l) (mapcar #'(lambda (x) (* 2 x)) l)) (app '(1 2)) (app '(2 3)) @@ -468,12 +593,12 @@ This form returns the result of that formatter" " (alexandria:with-unique-names (col flet-args) `(let ((,col (make-appender))) - (flet ((,name (&rest ,flet-args) - (if ,flet-args - (funcall ,col (apply (lambda ,fn-args ,@fn-body) - ,flet-args)) - (funcall ,col)))) - ,@body)))) + (map-aggregation (,col (lambda ,fn-args ,@fn-body)) + (flet ((,name (&rest ,flet-args) (apply ,col ,flet-args))) + ,@body))))) + + + ;; Copyright (c) 2002-2006, Edward Marco Baringer ;; 2011 Russ Tyndall , Acceleration.net http://www.acceleration.net diff --git a/tests/collectors.lisp b/tests/collectors.lisp index e47b0c9..6ac9b0d 100644 --- a/tests/collectors.lisp +++ b/tests/collectors.lisp @@ -16,6 +16,36 @@ (assert-eql 9 (r 1 2 3)) (assert-eql 9 (r)))) +(define-test simple-collector-test (:tags '(simple collector)) + (let* ((c (make-simple-collector))) + (assert-equal '(1 2 3 4 (5)) (funcall c 1 2 3 4 '(5))) + (assert-equal '(1 2 3 4 (5)) (funcall c)) + (assert-equal '(1 2 3 4 (5) 6) (funcall c 6)))) + +(define-test simple-collector-to-place-test (:tags '(simple collector place)) + (let* ((p) + (c (make-simple-collector-to-place p))) + (assert-equal '(1 2 3 4 (5)) (funcall c 1 2 3 4 '(5))) + (assert-equal '(1 2 3 4 (5)) (funcall c)) + (assert-equal '(1 2 3 4 (5)) p) + (assert-equal '(1 2 3 4 (5) 6) (funcall c 6)) + (assert-equal '(1 2 3 4 (5) 6) p))) + +(define-test simple-appender-test (:tags '(simple appender)) + (let* ((c (make-simple-appender))) + (assert-equal '(1 2 (3) 4 5) (funcall c 1 '(2 (3)) 4 '(5))) + (assert-equal '(1 2 (3) 4 5) (funcall c)) + (assert-equal '(1 2 (3) 4 5 6 7) (funcall c '(6 7))))) + +(define-test simple-appender-to-place-test (:tags '(simple appender place)) + (let* ((p) + (c (make-simple-appender-to-place p))) + (assert-equal '(1 2 (3) 4 5) (funcall c 1 '(2 (3)) 4 '(5))) + (assert-equal '(1 2 (3) 4 5) (funcall c)) + (assert-equal '(1 2 (3) 4 5) p) + (assert-equal '(1 2 (3) 4 5 6 7) (funcall c '(6 7))) + (assert-equal '(1 2 (3) 4 5 6 7) p))) + (define-test with-collector-test (:tags '(collector)) (with-collector (test) (test :a nil :key) @@ -107,17 +137,15 @@ (assert-equal "0 0 0-1-2-3-ABC" (test)))) (define-test with-mapping-collector-test (:tags '(mapping-collector collector)) - (with-mapping-collector (test (&rest nums) - (apply #'+ nums)) + (with-mapping-collector (test (x) (* 2 x)) (test 1) (test 1 2) (test 1 2 3) (test 1 2 3 4) - (assert-equal '(1 3 6 10) (test)))) + (assert-equal '(2 2 4 2 4 6 2 4 6 8) (test)))) (define-test with-mapping-appender-test (:tags '(mapping-appender appender)) - (with-mapping-appender (test (&rest nums) - (mapcar (lambda (x) (* 2 x)) nums)) + (with-mapping-appender (test (x) (* 2 x)) (test 1) (assert-equal '(2) (test)) (test 1 2) @@ -162,3 +190,16 @@ (assert-equal '(1 2 4 5) (deoperate c 3)) (assert-equal '(1 5) (deoperate c '(2 4 6 8))))) +(define-test nested-collectors (:tags '(collector)) + (with-collector (x) + (with-mapping-collector (z (it) (if it + (* 2 it) + (invoke-restart 'collectors-signals:skip))) + (with-collector (y :collect-nil nil) + (assert-equal '(1 nil 2) (x 1 nil 2)) + (assert-equal '(1 2) (y 1 nil 2)) + (assert-equal '(2 4) (z 1 nil 2)) + (assert-equal '(1 2 1 2) (y 1 nil 2)) + (assert-equal '(2 4 2 4) (z 1 nil 2)) + (assert-equal '(1 nil 2 1 nil 2) (x 1 nil 2)) + ))))