Advantages over .ps1 script

[jborean93]

I'm hoping to figure out what you see as the advantages of this tool over just having a pre-canned set of functions that someone can dot source. I have some idea as to what I think this really should be which I'll go into below but I first wanted to ask that question to see if I'm missing anything.

Using the ls example here is what I would see as a very basic copy as a script

Function Invoke-ls {
    [CmdletBinding()]
    param (
        [Parameter(Position=0)]
        $Path = '.',
 
        [Switch]
        $Detail,
 
        [Switch]
        $Item,
 
        [Switch]
        $OmitGroup,
 
        [Switch]
        $OmitUser
    )
 
    $args = @($Path)
    if ($Detail) { $args += '-l' }
    if ($Item) { $args += '-d' }
    if ($OmitGroup) { $args += '-g' }
    if ($OmitUser) { $args += '-o' }
 
    ls @args        
}

Anyone who has worked with PowerShell should be able to understand what's happening there and they can easily tweak certain things based on what their desired behaviour is. If we compare that to the json representation we effectively are writing the function as as an object and replicating things like parameter declarations, defaults and so on. This in my mind is just duplication for little gain and you've now added these disadvantages:

• No comments
  • Or at least the blog indicates that is the case, pwsh 7.1 seems to read json with comments just fine
• A new structure syntax that people need to be aware of
  • It seems to mimick very closely the param and cmdlet definitions in PowerShell, why duplicate this into another format
• Rigidity, can only configure what the json tool can expose
  • What happens when you have a tool that requires some special handling that the build tool can't handle properly, you now have to go back to a script to handle it all
• You are stuck with json conventions
  • Long strings are crazy, people will start using aliases - see blog post
  • You need to escape " and \ making it less of a 1 for 1 representation
• No intellisense, the code is json and you loose out of things like suggestions, code formatting, highlighting, etc
  • This makes it harder to debug and ultimately harder to maintain
• You cannot debug json, you would have to generate the scripts with crescendo first and then run that
  • Probably a given based on other alternatives but a pure native PowerShell method could mean they are debugable straight away without compilation

Looking at the example for apt in the blog post we can see the output handler is the following

"Handler": "$args[0]| select-object -skip 1 | %{$n,$v,$p,$s = "$_" -split ' '; [pscustomobject]@{ Name = $n -replace '/now'; Version = $v; Architecture = $p; State = $s.Trim('[]') -split ',' } }"
        }

De json-ifying that it now becomes

$args[0]| select-object -skip 1 | %{$n,$v,$p,$s = "$_" -split ' '; [pscustomobject]@{ Name = $n -replace '/now'; Version = $v; Architecture = $p; State = $s.Trim('[]') -split ',' } }

To me that's quite ugly and honestly going to be hard to maintain going forward. Compare that to a psd1 file

@{
    OutputHandlers = @(
        @{
            ParameterSetName = 'Default'
            Handler = {
                $args[0] | 
                  Select-Object -Skip 1 |
                  ForEach-Object {
                      $n, $v, $p, $s = "$_" -split ' '
                      [pscustomobject]@{
                          Name = $n -replace '/now'
                          Version = $v
                          Architecture = $p
                          State = $s.Trim('[]') -split ','
                    }
                }
            }
        }
    )
}

While I still think this is too nested and the benefit would be more suited towards writing scripts at least the handler code is IMO more readable than what was in the json. I see the 2nd example has a multiline json string and it does seem like PowerShell can parse it but it doesn't seem to conform to the json standard that most tools work with. This makes it harder for external tools to generate these files in a human readable fashion.

What I personally think would be a better way is to make Crescendo a collection of scripts that wrap native tools with some boilerplate functions that handle all the native calls and handling of errors. An example of what I had in mind using ls as an example would be

class Crescendo : Attribute {
    [String]$Name

    Crescendo([string]$Name) {
        $this.Name = $name
    }
}

Function Invoke-ls {
    [CmdletBinding()]
    param (
        [Parameter(Position=0)]
        $Path = '.',
 
        [Switch]
        [Crescendo('-l')]
        $Detail,
 
        [Switch]
        [Crescendo('-d')]
        $Item,
 
        [Switch]
        [Crescendo('-g')]
        $OmitGroup,
 
        [Switch]
        [Crescendo('-o')]
        $OmitUser
    )

    # Through the PSCmdlet we can get
    #     1. The binary to invoke from $PSCmdlet.MyInvocation.MyCommand.Noun
    #     2. The parameter set from $PSCmdlet.MyInvocation.Parameters
    #         Each parameter also cac access the OriginalName attribute which contains what they map to
    #     3. The values for each parameter with $PSCmdlet.MyInvocation
    Invoke-Crescendo $PSCmdlet

    # If the output needs to be massage then the function can do whatever needs to be done
}

Obviously this is very bare bones and not entirely thought out but I think it's definitely possible to to leave a common function structure that Crescendo can parse and automatically invoke for you while still giving users the flexibility to go against the grain if need be as well as use the existing syntax and tooling available to PowerShell today.

[Jaykul]

In fact, the example json for apt in the blog post is actually invalid, because they didn't escape the double quotes. A perfect example of why json is not a good place to store code...

However, the command does handle comments fine, and even handles multi-line strings which are NOT valid json either.

[Jaykul]

I feel like you're misunderstanding Crescendo. Crescendo uses the json to power / configure a PowerShell code generator which generates modules ... which don't have any dependencies. I think that's the right way to do native tool wrappers. We don't want a "native tool wrapper" module that has to actually dynamically generate code on the fly or anything like that -- it would be too slow.

However, the json configuration document doesn't seem great. I love the idea of using actual PowerShell syntax to write the template instead of json, but it should still just be a template to drive the generator. Maybe we could write the equivalent configuration for apt like this:

function Get-InstalledPackage {
    [Alias("apt")]
    [CmdletBinding()]
    param()

    Set-Handler @{
        Default = @{ 
            StreamOutput = $False
            Handler = {
                $args[0] | Select-Object -skip 1 | ForEach-Object {
                    $n, $v, $p, $s = "$_" -split ' '
                    [PSCustomObject]@{
                        Name = $n -replace '/now'
                        Version = $v
                        Architecture = $p
                        State = $s.Trim('[]') -split ','
                    }
                }
            }
        }
    }

    apt -q list --installed
}

[jborean93]

And while this is not a definite truth for all binaries, using ones like apt as an example. The time it takes for apt to run will generally make the time it takes for PowerShell to run the code inconsequential.

[Jaykul]

That's what they all say 😉 ... and then they parse 16000 lines of apt results and generate a psobject for each one, and have PowerShell's formatter re-generate the string output...

[jborean93]

But that’s not something a pre-generated script like Crescendo would fix? Both scenarios would still have to process the output at runtime, both have a scriptblock which would run, it’s just a matter of how it’s defined.

[sharpninja]

This is screaming to be refactored with YAML which would make it easy to include code snippets.

[gaelcolas]

I think it should maybe not accept JSON but just object ([IDictionary]), and then we can feed it what we want (JSON, Yaml, PSD1).
The JSON schema then would be useful for JSON and Yaml, but if someone prefer PSD1, that'd work too (without validation and suggestions).

[JamesWTruher]

It's definitely the case that the whole output handler thing needs to be worked over. I wasn't satisfied when I wrote it (the whole multi-line thing, as well as other things), but I'm still trying to figure out a way to improve it. It works best when you can easily convert the output with something like convertfrom-json. Maybe a pointer to a file, or function (but that rather weakens the whole declarative nature), I'm still noodling on it. However, it can just emit the text output, you still get the pipeline parameter binding, and tab-completion, etc.

Crescendo does happen to handle some trickier things as well (take a look at the dd wrapper) which may or may not be a problem for you. I chose json for a couple of reasons. One, I rather like the declarative rather than the code approach and more importantly, being able to use a schema when authoring. Currently, I'm not aware of a way to produce annotated schemas for psd1 files (I've thought about how to do this too).

my earliest demos actually did create the functions dynamically in the session, ala:

PS> invoke-expression ((Import-CommandConfiguration foo.crescendo.json).ToString())

but this has obvious problems.

It's also possible that Crescendo can also be used as an accelerator, it would be fairly simple to scaffold a bunch of commands and tweak them slightly post Export-CrescendoModule. If you're able to write your own wrappers, you probably don't need this at all.

Please note that this is a 0.4.0 version, we're still very early days.

[Jawz84]

Thanks for sharing your ideas and considerations @JamesWTruher. I realise it is early days.

I had doubts when I saw the powershell code wrapped in JSON. I don't think we should embed the handler code that way. No intellisense, escaping needed, no syntax highlighting, no static analysis etc.

I already suspected json was chosen because of the schema support it offers, and I know psd1 files do not offer schema support. I agree schema support has a lot of advantages. For instance VSCode has great json schema support. Schema enables key completion and config validation etc. And json does enable a declarative way of writing the config enforced by schema.

You arethinking about options to improve the output handler.
Options I can think of:

1. Create some form of schema support for psd1 and make psd1 the config format.
2. Leave the powershell code out of the json, just provide a reference to the script file.
3. Create a DSL instead, declarative in nature.

Each have their own drawbacks.

1. Even if psd1 got schema support, that doesn't mean VSCode/powershell extension supports it as a first class citizen immediately.
2. Json schema does not provide a way to validate a path. It is not filesystem aware. The config is no longer all in one place, making it easier to overlook mismatches etc.
3. Needs designing a whole DSL for this.

That brings up the question: what are the end goals here. Or better: who is the target audience, and what do they need?

Users who can't (be bothered to) write native command wrappers themselves, but would like to have tab completion and object output etc.

On Windows, a lot of ground is already covered with the Windows Powershell family of modules. On Mac/Linux is where there less modules available for tasks, and thus where more native commands need to be used.

We are targeting the 'enablers' here, the people who understand a bit more, and want to help make other peoples lives easier. They are not the one struggling with powershell, because they already understand the power of object based output, tab completion etc. Otherwise they would not miss that on native commands.

We want something that is easy to work with, and helps users to fall in the pit of success, to quote from the code standards.

Having considered all this, I think for this audience, it is fine having the json with a schema, and a reference to a powershell script file and some domuntation and examples like there already are here.
This makes it easy to create a valid json config, and makes it easy to write a valid handler, because both are edited in their natural habitat, with all tools available.

[SeeminglyScience]

I chose json for a couple of reasons. One, I rather like the declarative rather than the code approach and more importantly, being able to use a schema when authoring. Currently, I'm not aware of a way to produce annotated schemas for psd1 files (I've thought about how to do this too).

Something to consider as an alternative is a DSL. You can still have it be declarative by operating on AST traversal instead of actual invocation, while also providing rich editing experience.

[russellds]

I don't mind JSON being a declarative way of defining the function, it is simple enough, the only hang up I have is the Output Handler being in JSON. I think having it provide a path to a .ps1 file would be much easier to troubleshoot and support. As far as the .ps1 file is concerned, for ease of testing, I recommend that the .ps1 file would be a function that has a single parameter of InputObject. That way it matches the current behavior but allows for testing the function outside of Crescendo. Then Crescendo could just include the function inside of the generated function. To me, this would be the simplest way to solve this issue while still leveraging JSON.

[theJasonHelmick]

@iSazonov - I'll add some comments to this...

1. Who the target audience of this tool is? Who are these people?
   Anyone that wishes to quickly wrap a native command that is not familiar or comfortable with API wrapping. A large portion of PowerShell users don't have those skills - but would benefit from being able to quickly wrap a native command making it easier and more comfortable to use in their automation.
2. How widely will this tool be used?
   It's early at this stage and there are some challenges (such as output handling) that limit the adoption at this point, not to mention its in preview. I think this would serve a variety of audiences, including developers that may wish to have a quick way to wrap native commands and reduce maintenance over time, but I see most of this benefiting the IT admin.
3. Do you think MSFT teams will generate wrappers with the tool for most Windows utilities and distribute its with Windows?
   We have an internal team that is using parts of Crescendo today, but we are still in preview and have some challenges to address. Would other teams adopt this to wrap native commands that they produce? I'm discussing it.;)
4. Do you think third-party Linux vendors (Redhat, Ubuntu) and Apple will generate wrappers with the tool for most Unix utilities?
   My goal was to make this easy enough that a Powershell user would feel comfortable wrapping whatever native command they want. While it would be nice to think that 3rd parties would pre-Crescendo native commands for us, that's not my expectation.
5. Do you think third-party developer teams (C#, C++, C, Rust and so on) will generate wrappers with the tool for their utilities?
   Same as above - It's certainly possible.

[iSazonov]

@theJasonHelmick Thanks!

We have an internal team that is using parts of Crescendo today, but we are still in preview and have some challenges to address. Would other teams adopt this to wrap native commands that they produce? I'm discussing it.;)

If MSFT is going to invest in this area, it is much more productive to have generic classes like CommandLineParser project so that the exe contains metainformation that PowerShell can use directly and in a standard way. I can imagine that developers will use such standard classes as a best practice since they need a parser anyway, but they are unlikely to take the time to create intricate entities for any shells.

[ninmonkey]

ConvertFrom-Json can read files with comments in PS6+

[Jaykul]

And apparently, multi-line strings -- although I can't figure out how to make VS Code not complain about those.

[Halkcyon]

@Jaykul They're not spec-compliant, so I doubt you can make your editor stop complaining about them.

[JustinGrote]

YAML and TOML have the same json schema support as JSON :)

[russellds]

But neither YAML or TOML have native support from PowerShell.

[Jaykul]

But neither YAML or TOML have native support from PowerShell.

So what? 😛

[vexx32]

And also, I don't really know why JSON is the first option, when it probably would have been just as useful (arguably possibly more useful) to add schema support to PSD1 files, even if said schema enforcement is an additional module that needs installed.

[jcotton42]

A point being brought up in favor of JSON seems to be schema support.

While that certainly is a benefit, as a Crescendo file author I feel I would get much more out of not having to fight with quoting, lack of syntax highlighting, lack of IntelliSense, etc. than having a schema.

[michaeltlombardi]

It also implies adding schema support for PowerShell data files is a good idea - or we could skip that functionality (initially) with a Test-CrescendoManifest function a la Test-ModuleManifest.

[michaeltlombardi]

Speaking as someone responsible for maintaining CLI tools (and likely one of the people on the hook for "PowerShellizing" tools at Puppet I don't directly own), an implementation leveraging PowerShell classes, functions, or data files with a strongly opinionated snippet and a validation function would be more useful, debuggable, and maintainable for me than writing PowerShell in a JSON string.

It's fine if the first (several!) iterations of the tooling are limited in capability and expanded on over time; the maintainability of the definitions is the primary concern for this imo. The fewer levels of interpolation required, the better.

[Halkcyon]

The goal is not to write PowerShell. The PowerShell gets generated from the schema.

[michaeltlombardi]

Except that the Handler code in OutputHandlers will always be PowerShell, right? And then we're managing writing PowerShell in a JSON string.

[poke]

In my comment on issue 27 I outlined the problems I see with using JSON and also suggested a PowerShell-native approach which likely would be more powerful and readable. See the issue for details but here is what the ipconfig example could look like:

$module = New-CrescendoModule -Verb Get -Noun IpConfig `
    -OriginalName "c:/windows/system32/ipconfig.exe" `
    -Description "This will display the current IP configuration information on Windows"

Add-CrescendoParameter $module -Name All -OriginalName "/all" -ParameterType Switch `
    -Description "This switch provides all ip configuration details"

Add-CrescendoParameter $module -Name AllCompartments -OriginalName "/allcompartments" -ParameterType Switch `
        -Description "This switch provides compartment configuration details"

Add-CrescendoOutputHandler $module -ParameterSetName Default -Handler {
    param ( $lines )
    $post = $false;
    foreach($line in $lines | ?{$_.trim()}) {
        $LineToCheck = $line | select-string '^[a-z]';
        if ( $LineToCheck ) {
            if ( $post ) { [pscustomobject]$ht |add-member -pass -typename $oName }
            $oName = ($LineToCheck -match 'Configuration') ? 'IpConfiguration' : 'EthernetAdapter';
            $ht = @{};
            $post = $true
        }
        else {
            if ( $line -match '^   [a-z]' ) {
                $prop,$value = $line.split(' :',2);
                $pName = $prop -replace '[ .-]';
                $ht[$pName] = $value.Trim()
            }
            else {
                $ht[$pName] = .{$ht[$pName];$line.trim()}
            }
        }
    }
    [pscustomobject]$ht | add-member -pass -typename $oName
}

Export-CrescendoModule $module -ModuleName Ipconfig.psm1

[gaelcolas]

You know you can do something like this already right?
You can directly use the Object model and change the classes properties before exporting the module.

It's not as thinly wrapped as you describe above, but you can do something similar already.

[ethanbergstrom]

To add to what @gaelcolas mentioned, here's a very basic example I got working that still allows the Handler scriptblock to be subject to Intellisense, syntax highlighting, and handles string escaping for you automatically.

$tempJson = New-TemporaryFile

New-CrescendoCommand -Verb 'Get' -Noun 'ChocoPackage' | %{
    $_.OriginalName = 'choco'
    $_.OriginalCommandElements = @(
        'search',
        '--limit-output',
        '--local-only'
    )
    $_.Description = 'Get installed Chocolatey packages'
    $_.Parameters = @(
        @{
            Name = 'Package'
            ParameterType = 'string'
            Description = 'Package Name'
        }
    )
    $_.OutputHandlers = @(
        @{
            ParameterSetName = 'Default'
            Handler = {
                param ( $output )
                $output | ForEach-Object {
                    $name,$version = $_ -split '\|'
                    [pscustomobject]@{
                        Name = $name
                        Version = $version
                    }
                }
            }
        }
    )
    $_
} | ConvertTo-Json | Out-File $tempJson

Export-CrescendoModule -ConfigurationFile $tempJson -ModuleName .\Get-ChocoPackage.psm1 -Force

What's really nice about this approach, is if multiple commands use the same output handler logic, you can store that scriptblock separately and reference multiple times.

Though it also would be nice to just take the Command object and directly export it, which would skip the whole JSON step completely...

Edit: Iterated on this a bit further to dynamically construct and inherit command data by looping through a nested array of Nouns and Verbs with original element and parameter data at multiple levels. See here for an example.

[majkinetor]

This is the first thing that crossed my mind - why json when we can have HashTable like champions? If this is for semi-n00b users, then schema will not help much, and if it is really going to be that complex to need schema, doesn't it defeats the end goal ?

Why not supporting HashTable configuration besides the json ? For me at least, writing a PowerShell handler in json is instant turndown. This could ofc. be done externally as converter but it would be more stable to support it OTB.

[theJasonHelmick]

@majkinetor Thanks for the feedback! There are two reasons we chose this over say psd1 or other options. 1. It's schematized to assist in editing (even new users need the help) and 2. For modern commands that emit structured data with their output and sometimes help output, it may be something we could automate easily and build the json in some cases. Regardless if autogenerated or not, there are many times I imaging someone wanting to adjust/edit/change the json before generating.

[jcotton42]

In my opinion having syntax highlighting, IntelliSense, and not having to fight with quoting vastly outweighs any benefits a schema might bring to editing.

And maybe this just means we could use psd1 schemas.

[theJasonHelmick]

@jcotton42 Thank for your opinion! today, psd1's do not support schemas which is why we didn't use them.

[vexx32]

That's sort of the point. Schema support would be beneficial beyond just this module, but would definitely make using this module easier (if it supported PSD1s, and had some form of schema support for them).

Seems like a no-brainer to add schema support.

[JustinGrote]

While not all commands would have this, it would make sense to at least first use a bash completer, use https://github.com/PowerShell/Modules/tree/master/Modules/Microsoft.PowerShell.UnixCompleters to convert it into a usable C# type model, and then use that model to autogenerate a crescendo json.

[ghost]

Anything but not JSON, Please.