diff --git a/CHANGELOG.md b/CHANGELOG.md index 5f47e176..61bc716e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,122 @@ ## HEAD +## [reflow0.6](https://github.com/grailbio/reflow/releases/tag/reflow0.6) + +This release introduces many improvements to Reflow's user experience +and performance and, as always, many bug fixes and smaller +improvements. + +The major changes in this release are: *bottom up evaluation*, which +allows Reflow to use both logical and physical cache keys in order to +improve cache utilization and hit rates; *lazy object fetching* which +minimizes the amount of data transfer Reflow needs to perform from +cache; and *status displays* which, when Reflow is run in a capable +terminal, displays current evaluation status to the user, so that +it's always simple to understand what Reflow is doing at a glance. +Many of Reflow's commands are also clarified and simplified. For +example "reflow info" is now capable of displaying information about +any valid Reflow identifier (e.g., a cached object, a running exec, +an alloc, or a cached fileset). + +Reflow 0.6 also introduces some changes in how cache entries are +stored. This is in order to support abbreviated lookups of cache +entries (e.g., "reflow info abcdef"). Command "reflow migrate" +migrates the current database setup to support this feature. + +The changes and improvements are detailed below. + +### Core + +- [bottom-up evaluation](https://github.com/grailbio/reflow/commit/2093b25253069dc99e996a3e4080ef522d783080): + With "reflow run -eval=bottomup", reflow + performs pure, cache-assisted bottom-up evaluation. This allows + Reflow to compute physical cache keys in addition to logical ones, + and can thus (safely) cache more widely. Bottom-up evaluation logs + also never skip any part of the evaluation, thus providing a more + sensible program trace. In bottom-up mode, Reflow looks up both + keys. +- precise, lazy object fetching: Reflow now fetches objects from + cache by need, and not when the corresponding cache entry is + retrieved. This means that Reflow now never fetches more data + than is absolutely required. +- many improvements to evaluation debug logging (reflow -log debug run) +- add configuration provider for static clusters +- extensible resource types (beyond cpu, mem, disk) +- use pure concurrency limits for transfers +- a number of cluster scheduler tweaks and improvements +- add a [status reporting](https://github.com/grailbio/reflow/commit/9568976b561eb35a14f2923b60f70ddcb814b810) + facility that is used to report the current set of evaluation activities + (executions, interns, data transfers, cluster management activities) to the user in + a comprehensible way +- identifier-based invalidation: 'reflow run -invalidate=regexp ..." will invalidate + cache entries produced by nodes whose identifiers match the provided regular + expression. For example, the command "reflow run -invalidate=[aA]lign wgs.rf ..." + will run wgs.rf as normal, but recompute all alignments. + +### Tool + +- unified handling of sha256s: "reflow info" can now resolve + any valid Reflow identifier +- "reflow info" can now resolve sha256 prefixes; this requires adding + indices to DynamoDB table with new command "reflow migrate" +- command "reflow cat" writes a cached object to stdout +- command "reflow sync" synchronizes a cached fileset to a local destination +- "reflow doc" displays all available system modules when invoked without + arguments +- "reflow doc" now renders all default parameters by pretty-printing their + expressions in an abbreviated form +- handle interrupts (control-c) properly, so that allocs can be reclaimed + much more quickly after an interrupted run + +### Syntax + +- [comprehension support](https://github.com/grailbio/reflow/commit/f8f0b300d9dc98fe88a8464fe3249409a51dc4f6) + for cross products and filters +- $/regexp.Match: regular expression matching +- $/regexp.Replace: regular expression matching with replacement +- $/strings.FromFloat: convert floating point numbers to strings +- $/strings.Join, $/strings.HasPrefix, $/strings.HasSuffix: string utilities +- support missing arithmetic binary operations for ints and floats: + %, <<, >>, -, / +- improve error messages in various standard library funcs +- various improvements to type and syntax error messages +- add registration mechanism for custom intrinsics +- support for expressing cpu features in resource requirements + (e.g., "intel_avx", "intel_avx2", and "intel_avx512" as supported by various + EC2 instance types) +- [trace builtin](https://github.com/grailbio/reflow/commit/dfcbf744852276e8f6d6ac912c26c4d7b139a515) + to aid debugging +- support fractional CPUs in resource requirements +- [range builtin](https://github.com/grailbio/reflow/commit/9f09974ead5e4748aa013fb9e13e09ec236c00c3): + create integer ranges +- support rich(er) key types in maps: string, int, float, bool, and file are now supported + +### Cache + +- record latest access times counts to permit for LRU collection +- support digest abbreviations + +### Local executor + +- don't attempt to hardlink across devices +- more careful about parsing Docker repository names to avoid + unnecessary pulls + +### ec2cluster + +- update definitions from Amazon +- change resource fudge factor; instance-reported resource + availability changed with recent virtualization changes +- remove idle boundary computation no longer needed due to EC2 + billing changes; instead use a straightforward idle timeout +- [user-mergeable cloud-config](https://github.com/grailbio/reflow/commit/f1768adc0915821dbb7c5efb6e9a6f71a6347f3a) +- instances launched by ec2cluster may now be assigned a (configurable) + EC2 instance profile +- batch DescribeInstance calls with paging to reduce EC2 API load +- support for avx, avx2, and avx512 resource constraints +- allocation improvements resulting in better instance packing +- configurabhle [striping of multiple EBS volumes](https://github.com/grailbio/reflow/commit/53696ed3c3f7aba305dc38945118dba9e85f4c99) + to improve throughput; with this change we can push >2GiB/s on larger instance types ## [reflow0.5.3](https://github.com/grailbio/reflow/releases/tag/reflow0.5.3) diff --git a/cmd/reflow/version.go b/cmd/reflow/version.go index 680bbc16..304c99d3 100644 --- a/cmd/reflow/version.go +++ b/cmd/reflow/version.go @@ -1,12 +1,8 @@ -// Copyright 2018 GRAIL, Inc. All rights reserved. -// Use of this source code is governed by the Apache 2.0 -// license that can be found in the LICENSE file. - // This file is automatically generated by releasereflow. Do not edit! package main const ( - version = "reflow0.5.3" - reflowlet = "grailbio/reflowlet:1510608135" + version = "0.6" + reflowlet = "grailbio/reflowlet:1519329028" )