Skip to content

Commit

Permalink
revise: multiple improvements to TES backend
Browse files Browse the repository at this point in the history
* Crankshaft to TES data structure conversion is now done via the `From`
  trait.
* Initial gathering of log data is reenabled.
* Some slight changes to logging.
  • Loading branch information
claymcleod committed Feb 10, 2025
1 parent 254601c commit c0f49ef
Show file tree
Hide file tree
Showing 13 changed files with 320 additions and 124 deletions.
70 changes: 43 additions & 27 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 1 addition & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ members = [
"crankshaft-engine",
"examples",
]
default-members = ["crankshaft"]
resolver = "2"

[workspace.package]
Expand Down Expand Up @@ -39,7 +38,7 @@ shlex = "1.3.0"
ssh2 = "0.9.4"
tar = "0.4.41"
tempfile = "3.12.0"
tes = { version = "0.3.0", features = ["client", "serde"] }
tes = { version = "0.4.0", features = ["client", "serde"] }
thiserror = "2.0.11"
tokio = { version = "1.40.0", features = ["full", "time", "tracing"] }
tokio-metrics = "0.3.1"
Expand Down
3 changes: 2 additions & 1 deletion crankshaft-config/src/backend.rs
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ impl Config {
self.defaults.as_ref()
}

/// Consumes `self` returns the constituent parts of the [`Config`].
/// Consumes `self` returns the constituent, owned parts of the
/// configuration.
pub fn into_parts(self) -> (String, Kind, usize, Option<Defaults>) {
(self.name, self.kind, self.max_tasks, self.defaults)
}
Expand Down
8 changes: 7 additions & 1 deletion crankshaft-config/src/backend/tes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ pub struct Config {
url: Url,

/// More nuanced, HTTP-related configuration.
#[builder(into)]
#[builder(into, default)]
http: http::Config,
}

Expand All @@ -31,4 +31,10 @@ impl Config {
pub fn http(&self) -> &http::Config {
&self.http
}

/// Consumes `self` and returns the constituent, owned parts of the
/// configuration.
pub fn into_parts(self) -> (Url, http::Config) {
(self.url, self.http)
}
}
2 changes: 2 additions & 0 deletions crankshaft-engine/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
([#8](https://github.com/stjude-rust-labs/crankshaft/pull/8)).
* Adds `Resource::builder()` and `Output::builder()` to match the
`Input::builder`.
* Multiple revisions to the TES backend
([#9](https://github.com/stjude-rust-labs/crankshaft/issues/9)).
* Better handling for URL contents in inputs.
* Swaps out most of the bespoke builders for `bon`.
* Removes `#[builder(into)]` for numerical types
Expand Down
2 changes: 1 addition & 1 deletion crankshaft-engine/src/service/runner/backend/docker.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ impl crate::Backend for Backend {
// (2) Upload inputs to the container.
//
// TODO(clay): these could be cached.
for task in task.inputs().cloned().map(|i| {
for task in task.inputs().map(|i| {
let container = container.clone();
tokio::spawn(async move {
let contents = i.fetch().await;
Expand Down
Loading

0 comments on commit c0f49ef

Please sign in to comment.