Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose more configuration options for executor #1098

Closed
Tracked by #1068
milenkovicm opened this issue Oct 24, 2024 · 0 comments · Fixed by #1099
Closed
Tracked by #1068

Expose more configuration options for executor #1098

milenkovicm opened this issue Oct 24, 2024 · 0 comments · Fixed by #1099
Labels
enhancement New feature or request

Comments

@milenkovicm
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

On the road to library transition I would like to expose more configuration (DataFusion specific) which user can tune. This would enable us to remove hard coded object stores, codecs, configuration extension, functions ...

Describe the solution you'd like

I would like to expose additional method in executor/src/standalone.rs:

pub async fn new_standalone_executor_from_state<
    T: 'static + AsLogicalPlan,
    U: 'static + AsExecutionPlan,
>(
    scheduler: SchedulerGrpcClient<Channel>,
    concurrent_tasks: usize,
    session_state: &SessionState,
) -> Result<()> {

And optional parameter in executor/src/executor_process.rs:

pub struct ExecutorProcessConfig {
    ...
    /// SessionState which will be used to configure executor
    /// if not provided system will configure itself from 
    /// sensible defaults
    pub session_state: Option<SessionState>,
}

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context

  • Follow up commit would remove with_object_store_registry and BallistaObjectStoreRegistry giving user ability to plug their own and removing chunk of code from maintenance.
  • Config extension would be possible

depends on #1096

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant