Dynamic environment variables in the config file not working in ubuntu based build agents

[nikhil-neu]

Describe the bug
Env variables in config variables not working in ubuntu build agents .
Getting the error
Validation of config file completed. Update-PropertiesForObject: /root/.local/share/powershell/Modules/azure.datafactory.tools/0.95.1/private/Update-PropertiesFromFile.ps1:75 Line | 75 | … $null = Update-PropertiesForObject -o $_ -action $action -path $p … | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ | Cannot bind argument to parameter 'value' because it is an | empty string.

To Reproduce
Using this config file

Your script to reproduce the behaviour:

{
    "akv_ls_amnoneshd": [
      {
        "name": "$.properties.typeProperties.baseUrl",
        "value": "https://co-wus2-amnoneshd-kv-t01.vault.azure.net/",
        "action": "update"
      }
    ],
    "sql_ls_amie_amhdb": [
      {
        "name": "$.properties.connectVia.referenceName",
        "value": "ADFIntegrationRuntime",
        "action": "update"
      }
    ]
}

Expected behaviour
Config file should be used to paramterise the json files

Version of PowerShell module of: latest

• ver.........

[NowinskiK]

I run (under UBUNTU agent) the subset of unit tests included in the module and it works fine:

As an additional check I run this PS script under Ubuntu agent:

$m = 'azure.datafactory.tools'
Install-Module $m -Scope CurrentUser
Import-Module $m

$RootFolder = 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\BigFactorySample2'
$config = 'X:\!WORK\GitHub\!SQLPlayer\azure.datafactory.tools\test\BigFactorySample2\deployment\config-c100.json'

Publish-AdfV2FromJson -RootFolder $RootFolder -ResourceGroupName 'as' -DataFactoryName 'xyz' -Stage $config -DryRun

The above code finished successfully either, until empty value met, which is expected:

All previous configuration items have been processed properly.
Do double-check your values or variables. You can switch DEBUG mode ON to see more details.

[nikhil-neu]

@NowinskiK .. but my value isnt empty ..
Here is my config file
type,name,path,value linkedService,akv_ls_amnoneshd,typeProperties.baseUrl,"$Env:kvURI" linkedService,sql_ls_amie_amhdb,connectVia.referenceName,"intRunTimeAmnOneAmieDev"
i enabled verbose
`
Validation of config file completed.
VERBOSE: - Performing: update for object(path): linkedService.akv_ls_amnoneshd(properties.typeProperties.baseUrl)
VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: name type properties
VERBOSE: Processing property 'name' of type String
VERBOSE: Processing property 'type' of type String
VERBOSE: Processing property 'properties' of type PSCustomObject
VERBOSE: Converting PSCustomObject property using a recursive function call...
VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: annotations type typeProperties
VERBOSE: Processing property 'annotations' of type Object[]
VERBOSE: Processing property 'type' of type String
VERBOSE: Processing property 'typeProperties' of type PSCustomObject
VERBOSE: Converting PSCustomObject property using a recursive function call...
VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: baseUrl
VERBOSE: Processing property 'baseUrl' of type String
Update-PropertiesForObject: /root/.local/share/powershell/Modules/azure.datafactory.tools/0.95.1/private/Update-PropertiesFromFile.ps1:75
Line |
75 | … $null = Update-PropertiesForObject -o $_ -action $action -path $p …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Cannot bind argument to parameter 'value' because it is an
| empty string.

##[error]PowerShell exited with code '1'.
`

The same task can be passing here , when run on a windows agent with everything remaining same

VERBOSE: Properties: annotations type typeProperties
VERBOSE: Processing property 'annotations' of type Object[]
VERBOSE: Processing property 'type' of type String
VERBOSE: Processing property 'typeProperties' of type PSCustomObject
VERBOSE: Converting PSCustomObject property using a recursive function call...
VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: baseUrl
VERBOSE: Processing property 'baseUrl' of type String
VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: name type properties
VERBOSE: Processing property 'name' of type String
VERBOSE: Processing property 'type' of type String
VERBOSE: Processing property 'properties' of type PSCustomObject
VERBOSE: Converting PSCustomObject property using a recursive function call...
VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays
VERBOSE: Processing PSCustomObject...
VERBOSE: Properties: annotations type typeProperties
VERBOSE: Processing property 'annotations' of type Object[]
VERBOSE: Processing property 'type' of type String
VERBOSE: Processing property 'typeProperties' of type PSCustomObject
VERBOSE: Converting PSCustomObject property using a recursive function call...
VERBOSE: Entering Function: ConvertFrom-OrderedHashTablesToArrays
VERBOSE: Processing PSCustomObject...
VERBOSE: Processing property 'baseUrl' of type String
VERBOSE: - Performing: update for object(path): linkedService.sql_ls_amie_amhdb(properties.connectVia.referenceName)
VERBOSE: Entering Function: ConvertFrom-ArraysToOrderedHashTables
VERBOSE: Processing PSCustomObject...

[NowinskiK]

Can you attach both files of Linked Services?

[nikhil-neu]

@NowinskiK
akv_ls_amnoneshd

{
"name": "akv_ls_amnoneshd",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://co-wus2-xxxxxxx-kv-d01.vault.azure.net/"
}
}
}

sql_ls_amie_amhdb
{
"name": "sql_ls_xxxx_amhdb",
"type": "Microsoft.DataFactory/factories/linkedservices",
"properties": {
"annotations": [],
"type": "SqlServer",
"typeProperties": {
"connectionString": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "akv_ls_amnoneshd",
"type": "LinkedServiceReference"
},
"secretName": "xxxdb-sqlconnection-adf"
}
},
"connectVia": {
"referenceName": "intRunTimexxxOneAmieDev",
"type": "IntegrationRuntimeReference"
}
}
}

[nikhil-neu]

@NowinskiK ... any work around for this ?

[NowinskiK]

I'm on holiday right now. I will back to this as soon as I can.
In the meantime, you can investigate it on your own.

[nikhil-neu]

@NowinskiK .. i have been still unable to find a work around for this , would you be having ideas ?

[NowinskiK]

Unfortunately, I don't have too much capacity now, so we must ask someone from the community to investigate this issue deeper.

[NowinskiK]

I doubled-check everything today using my local machine with Visual Studio Code as well as Linux in Azure Cloud Shell.
The #adftools reports Cannot bind argument to parameter 'value' because it is an empty string. correctly.
The only difference between Windows & Linux is that Linux is case-sensitive.
If you define variable in Azure DevOps as kvURI it's represented as KVURI in environment variables.
Hence, your config file should be like the following one:

type,name,path,value
linkedService,akv_ls_amnoneshd,typeProperties.baseUrl,"$Env:KVURI"
linkedService,sql_ls_amie_amhdb,connectVia.referenceName,"intRunTimeAmnOneAmieDev"

To make sure whether the environment variable is set up - run printenv command:

This is how the variable is configured in Azure DevOps Build Pipeline: