Merge branch 'microsoft:main' into feature/workspace_user_management

.devcontainer/devcontainer.json

@@ -294,5 +294,5 @@
   ],
   // Run commands after the container is created.
   "postCreateCommand": "./.devcontainer/scripts/post-create.sh",
-  "initializeCommand": "mkdir -p $HOME/.azure $HOME/.config || true"
+  "initializeCommand": ["./.devcontainer/scripts/initialize"]
 }

.devcontainer/scripts/initialize

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+mkdir -p "$HOME/.azure" "$HOME/.config" || true

.devcontainer/scripts/initialize.cmd

@@ -0,0 +1,2 @@
+@echo off
+mkdir %USERPROFILE%\.azure %USERPROFILE%\.config || exit /b 0

.github/workflows/deploy_tre_reusable.yml

@@ -863,3 +863,4 @@ jobs:
         with:
           junit_files: "artifacts/**/*.xml"
           check_name: "E2E Test Results"
+          comment_mode: off

CHANGELOG.md

@@ -39,6 +39,7 @@ ENHANCEMENTS:
 * Add encryption at host for VMs ([#4263](https://github.com/microsoft/AzureTRE/pull/4263))
 * Downgrade certs shared service App Gateway to Basic SKU ([#4300](https://github.com/microsoft/AzureTRE/issues/4300))
 * Airlock function host storage to use the user-assigned managed identity ([#4276](https://github.com/microsoft/AzureTRE/issues/4276))
+* Disable local authentication in EventGrid ([#4254](https://github.com/microsoft/AzureTRE/issues/4254))
 
 BUG FIXES:
 * Update KeyVault references in API to use the version so Terraform cascades the update ([#4112](https://github.com/microsoft/AzureTRE/pull/4112))
@@ -56,7 +57,9 @@ BUG FIXES:
 * Fix VM actions where Workspace shared storage doesn't allow shared key access ([#4222](https://github.com/microsoft/AzureTRE/issues/4222))
 * Fix public exposure in Guacamole service ([[#4199](https://github.com/microsoft/AzureTRE/issues/4199)])
 * Fix Azure ML network tags to use name rather than ID ([[#4151](https://github.com/microsoft/AzureTRE/issues/4151)])
+* Windows R version must be 4.1.2 otherwise post install script doesn't update package mirror URL ([#4288](https://github.com/microsoft/AzureTRE/issues/4288))
 * Recreate tre_output.json if empty. ([[#4292](https://github.com/microsoft/AzureTRE/issues/4292)])
+* Ensure R directory is present before attempting to update package mirror URL ([#4332](https://github.com/microsoft/AzureTRE/pull/4332))
 
 COMPONENTS:
 

README.md

@@ -5,21 +5,23 @@
 ## Background
 <img align="right" src="./docs/assets/azure-tre-logo.svg" width="33%" />
 
-Across the health industry, be it a pharmaceutical company interrogating clinical trial results, or a public health provider analyzing electronic health records, there is the need to enable researchers, analysts, and developers to work with sensitive data sets.
+Across multiple industries, be it a pharmaceutical company interrogating clinical trial results, a public health provider analyzing electronic health records, or an engineering organization working on next-generation systems, there is a need to enable researchers and solution developers to collaborate on sensitive data.
 
-Trusted Research Environments (TREs) enable organisations to provide research teams secure access to these data sets alongside appropriate tooling to ensure researchers can remain efficient and productive despite the security controls in place.
+Trusted Research Environments (TREs) enable organisations to provide research and development (R&D) teams secure access to data alongside tooling to ensure productivity while keeping security controls in place.
 
-Further information on TREs in general can be found in many places, one good resource is [HDR UK's website](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/).
+Further information on TREs in general can be found in many places, one good resource specifically for healthcare is [HDR UK's website](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/).
+
+Azure TRE is finding new applications and to date, has been used as the basis for exploring future secure collaboration environments in engineering. Workloads investigated include systems and software development. In truth, any workload in any industry could theoretically be supported with extensions to Azure TRE.
 
 The Azure Trusted Research Environment project is an accelerator to assist Microsoft customers and partners who want to build out Trusted Research environments on Azure. This project enables authorized users to deploy and configure secure workspaces and researcher tooling without a dependency on IT teams.
 
-This project is typically implemented alongside a data platform that provides research ready datasets to TRE workspaces.
+This project is typically implemented alongside a data platform that provides research ready datasets to TRE workspaces. Azure TRE has also been proven to handle unstructured data scenarios. For example, collaborative Computer Aided Design (CAD), with Product Lifecycle Management integration.
 
 TREs are not “one size fits all”, hence although the Azure TRE has a number of out of the box features, the project has been built be extensible, and hence tooling and data platform agnostic.
 
 Core features include:
 - Self-service workspace management for TRE administrators
-- Self-service provisioning of research tooling for research teams
+- Self-service provisioning of R&D tooling for R&D teams
 - Package and repository mirroring - PyPi, R-CRAN, Apt and more.
 - Extensible architecture - build your own service templates as required
 - Microsoft Entra ID integration

airlock_processor/BlobCreatedTrigger/function.json

@@ -13,15 +13,13 @@
     {
       "type": "eventGrid",
       "name": "stepResultEvent",
-      "topicEndpointUri": "EVENT_GRID_STEP_RESULT_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_STEP_RESULT_TOPIC_KEY_SETTING",
+      "connection": "EVENT_GRID_STEP_RESULT_CONNECTION",
       "direction": "out"
     },
     {
       "type": "eventGrid",
       "name": "dataDeletionEvent",
-      "topicEndpointUri": "EVENT_GRID_DATA_DELETION_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_DATA_DELETION_TOPIC_KEY_SETTING",
+      "connection": "EVENT_GRID_DATA_DELETION_CONNECTION",
       "direction": "out"
     }
   ]

airlock_processor/ScanResultTrigger/function.json

@@ -12,8 +12,7 @@
     {
       "type": "eventGrid",
       "name": "outputEvent",
-      "topicEndpointUri": "EVENT_GRID_STEP_RESULT_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_STEP_RESULT_TOPIC_KEY_SETTING",
+      "connection": "EVENT_GRID_STEP_RESULT_CONNECTION",
       "direction": "out"
     }
   ]

airlock_processor/StatusChangedQueueTrigger/function.json

@@ -11,15 +11,13 @@
     {
       "type": "eventGrid",
       "name": "stepResultEvent",
-      "topicEndpointUri": "EVENT_GRID_STEP_RESULT_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_STEP_RESULT_TOPIC_KEY_SETTING",
+      "connection": "EVENT_GRID_STEP_RESULT_CONNECTION",
       "direction": "out"
     },
     {
       "type": "eventGrid",
       "name": "dataDeletionEvent",
-      "topicEndpointUri": "EVENT_GRID_DATA_DELETION_TOPIC_URI_SETTING",
-      "topicKeySetting": "EVENT_GRID_DATA_DELETION_TOPIC_KEY_SETTING",
+      "connection": "EVENT_GRID_DATA_DELETION_CONNECTION",
       "direction": "out"
     }
   ]

airlock_processor/_version.py

@@ -1 +1 @@
-__version__ = "0.8.0"
+__version__ = "0.8.1"

airlock_processor/run_tests_and_exit_succesfully.sh

@@ -6,6 +6,6 @@
 rm -f ../test-results/pytest_airlock_processor*
 mkdir -p ../test-results
 
-if ! pytest --junit-xml ../test-results/pytest_airlock_processor_unit.xml --ignore e2e_tests; then
+if ! python -m pytest --junit-xml ../test-results/pytest_airlock_processor_unit.xml --ignore e2e_tests; then
   touch ../test-results/pytest_airlock_processor_unit_failed
 fi

api_app/_version.py

@@ -1 +1 @@
-__version__ = "0.20.3"
+__version__ = "0.20.4"

api_app/run_tests_and_exit_succesfully.sh

@@ -6,6 +6,6 @@
 rm -f ../test-results/pytest_api*
 mkdir -p ../test-results
 
-if ! pytest --junit-xml ../test-results/pytest_api_unit.xml --ignore e2e_tests -W ignore::pytest.PytestUnraisableExceptionWarning -W ignore::DeprecationWarning; then
+if ! python -m pytest --junit-xml ../test-results/pytest_api_unit.xml --ignore e2e_tests -W ignore::pytest.PytestUnraisableExceptionWarning -W ignore::DeprecationWarning; then
   touch ../test-results/pytest_api_unit_failed
 fi

core/terraform/airlock/airlock_processor.tf

@@ -66,25 +66,31 @@ resource "azurerm_linux_function_app" "airlock_function_app" {
   }
 
   app_settings = {
-    "SB_CONNECTION_STRING"                       = var.airlock_servicebus.default_primary_connection_string
-    "BLOB_CREATED_TOPIC_NAME"                    = azurerm_servicebus_topic.blob_created.name
-    "TOPIC_SUBSCRIPTION_NAME"                    = azurerm_servicebus_subscription.airlock_processor.name
-    "EVENT_GRID_STEP_RESULT_TOPIC_URI_SETTING"   = azurerm_eventgrid_topic.step_result.endpoint
-    "EVENT_GRID_STEP_RESULT_TOPIC_KEY_SETTING"   = azurerm_eventgrid_topic.step_result.primary_access_key
-    "EVENT_GRID_DATA_DELETION_TOPIC_URI_SETTING" = azurerm_eventgrid_topic.data_deletion.endpoint
-    "EVENT_GRID_DATA_DELETION_TOPIC_KEY_SETTING" = azurerm_eventgrid_topic.data_deletion.primary_access_key
-    "WEBSITES_ENABLE_APP_SERVICE_STORAGE"        = false
-    "AIRLOCK_STATUS_CHANGED_QUEUE_NAME"          = local.status_changed_queue_name
-    "AIRLOCK_SCAN_RESULT_QUEUE_NAME"             = local.scan_result_queue_name
-    "AIRLOCK_DATA_DELETION_QUEUE_NAME"           = local.data_deletion_queue_name
-    "ENABLE_MALWARE_SCANNING"                    = var.enable_malware_scanning
-    "ARM_ENVIRONMENT"                            = var.arm_environment
-    "MANAGED_IDENTITY_CLIENT_ID"                 = azurerm_user_assigned_identity.airlock_id.client_id
-    "TRE_ID"                                     = var.tre_id
-    "WEBSITE_CONTENTOVERVNET"                    = 1
-    "STORAGE_ENDPOINT_SUFFIX"                    = module.terraform_azurerm_environment_configuration.storage_suffix
-    "AzureWebJobsStorage__clientId"              = azurerm_user_assigned_identity.airlock_id.client_id
-    "AzureWebJobsStorage__credential"            = "managedidentity"
+    "SB_CONNECTION_STRING"                = var.airlock_servicebus.default_primary_connection_string
+    "BLOB_CREATED_TOPIC_NAME"             = azurerm_servicebus_topic.blob_created.name
+    "TOPIC_SUBSCRIPTION_NAME"             = azurerm_servicebus_subscription.airlock_processor.name
+    "WEBSITES_ENABLE_APP_SERVICE_STORAGE" = false
+    "AIRLOCK_STATUS_CHANGED_QUEUE_NAME"   = local.status_changed_queue_name
+    "AIRLOCK_SCAN_RESULT_QUEUE_NAME"      = local.scan_result_queue_name
+    "AIRLOCK_DATA_DELETION_QUEUE_NAME"    = local.data_deletion_queue_name
+    "ENABLE_MALWARE_SCANNING"             = var.enable_malware_scanning
+    "ARM_ENVIRONMENT"                     = var.arm_environment
+    "MANAGED_IDENTITY_CLIENT_ID"          = azurerm_user_assigned_identity.airlock_id.client_id
+    "TRE_ID"                              = var.tre_id
+    "WEBSITE_CONTENTOVERVNET"             = 1
+    "STORAGE_ENDPOINT_SUFFIX"             = module.terraform_azurerm_environment_configuration.storage_suffix
+    "AzureWebJobsStorage__clientId"       = azurerm_user_assigned_identity.airlock_id.client_id
+    "AzureWebJobsStorage__credential"     = "managedidentity"
+
+    "EVENT_GRID_STEP_RESULT_CONNECTION"                           = local.step_result_eventgrid_connection
+    "${local.step_result_eventgrid_connection}__topicEndpointUri" = azurerm_eventgrid_topic.step_result.endpoint
+    "${local.step_result_eventgrid_connection}__credential"       = "managedidentity"
+    "${local.step_result_eventgrid_connection}__clientId"         = azurerm_user_assigned_identity.airlock_id.client_id
+
+    "EVENT_GRID_DATA_DELETION_CONNECTION"                           = local.data_deletion_eventgrid_connection
+    "${local.data_deletion_eventgrid_connection}__topicEndpointUri" = azurerm_eventgrid_topic.data_deletion.endpoint
+    "${local.data_deletion_eventgrid_connection}__credential"       = "managedidentity"
+    "${local.data_deletion_eventgrid_connection}__clientId"         = azurerm_user_assigned_identity.airlock_id.client_id
   }
 
   site_config {

core/terraform/airlock/eventgrid_topics.tf

@@ -6,6 +6,7 @@ resource "azurerm_eventgrid_topic" "step_result" {
   location                      = var.location
   resource_group_name           = var.resource_group_name
   public_network_access_enabled = var.enable_local_debugging
+  local_auth_enabled            = false
 
   identity {
     type = "SystemAssigned"
@@ -60,6 +61,7 @@ resource "azurerm_eventgrid_topic" "status_changed" {
   location                      = var.location
   resource_group_name           = var.resource_group_name
   public_network_access_enabled = var.enable_local_debugging
+  local_auth_enabled            = false
 
   identity {
     type = "SystemAssigned"
@@ -113,6 +115,7 @@ resource "azurerm_eventgrid_topic" "data_deletion" {
   location                      = var.location
   resource_group_name           = var.resource_group_name
   public_network_access_enabled = var.enable_local_debugging
+  local_auth_enabled            = false
 
   identity {
     type = "SystemAssigned"
@@ -163,6 +166,7 @@ resource "azurerm_eventgrid_topic" "scan_result" {
   resource_group_name = var.resource_group_name
   # This is mandatory for the scan result to be published since private networks are not supported yet
   public_network_access_enabled = true
+  local_auth_enabled            = false
 
   identity {
     type = "SystemAssigned"
@@ -323,6 +327,7 @@ resource "azurerm_eventgrid_topic" "airlock_notification" {
   location                      = var.location
   resource_group_name           = var.resource_group_name
   public_network_access_enabled = var.enable_local_debugging
+  local_auth_enabled            = false
 
   identity {
     type = "SystemAssigned"

core/terraform/airlock/identity.tf

@@ -25,7 +25,7 @@ resource "azurerm_role_assignment" "servicebus_receiver" {
   principal_id         = azurerm_user_assigned_identity.airlock_id.principal_id
 }
 
-resource "azurerm_role_assignment" "eventgrid_data_sender" {
+resource "azurerm_role_assignment" "eventgrid_data_sender_status_changed" {
   scope                = azurerm_eventgrid_topic.status_changed.id
   role_definition_name = "EventGrid Data Sender"
   principal_id         = var.api_principal_id
@@ -37,6 +37,18 @@ resource "azurerm_role_assignment" "eventgrid_data_sender_notification" {
   principal_id         = var.api_principal_id
 }
 
+resource "azurerm_role_assignment" "eventgrid_data_sender_step_result" {
+  scope                = azurerm_eventgrid_topic.step_result.id
+  role_definition_name = "EventGrid Data Sender"
+  principal_id         = azurerm_user_assigned_identity.airlock_id.principal_id
+}
+
+resource "azurerm_role_assignment" "eventgrid_data_sender_data_deletion" {
+  scope                = azurerm_eventgrid_topic.data_deletion.id
+  role_definition_name = "EventGrid Data Sender"
+  principal_id         = azurerm_user_assigned_identity.airlock_id.principal_id
+}
+
 resource "azurerm_role_assignment" "airlock_blob_data_contributor" {
   count                = length(local.airlock_sa_blob_data_contributor)
   scope                = local.airlock_sa_blob_data_contributor[count.index]

core/terraform/airlock/locals.tf

@@ -60,4 +60,7 @@ locals {
     azurerm_storage_account.sa_import_in_progress.id,
     azurerm_storage_account.sa_export_approved.id
   ]
+
+  step_result_eventgrid_connection   = "EVENT_GRID_STEP_RESULT_CONNECTION"
+  data_deletion_eventgrid_connection = "EVENT_GRID_DATA_DELETION_CONNECTION"
 }

core/version.txt

@@ -1 +1 @@
-__version__ = "0.11.21"
+__version__ = "0.11.22"

docs/index.md

@@ -2,22 +2,26 @@
 
 ## What is Azure TRE?
 
-Across the health industry, be it a pharmaceutical company interrogating clinical trial results, or a public health provider analyzing electronic health records, there is the need to enable researchers, analysts, and developers to work with sensitive data sets.  
+Across multiple industries, be it a pharmaceutical company interrogating clinical trial results, a public health provider analyzing electronic health records, or an engineering organization working on next-generation systems, there is a need to enable researchers and solution developers to collaborate on sensitive data.  
 
-Trusted Research Environments (TREs) enable organisations to provide research teams secure access to these data sets along side tooling to ensure a researchers can be productive. Further information on TREs in general can be found in many places, one good resource is [HDR UK](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/).
+Trusted Research Environments (TREs) enable organisations to provide research and development (R&D) teams secure access to data alongside tooling to ensure productivity. Further information on TREs in general can be found in many places, one good resource specifically for healthcare is [HDR UK](https://www.hdruk.ac.uk/access-to-health-data/trusted-research-environments/).
 
-The Azure Trusted Research Environment project is an accelerator to assist Microsoft customers and partners who want to build out Trusted Research environments on Azure. This project enables authorized users to deploy and configure secure workspaces and researcher tooling without a dependency on IT teams.  
+Azure TRE is finding new applications and to date, has been used as the basis for exploring future secure collaboration environments in engineering. Workloads investigated include systems and software development. In truth, any workload in any industry could theoretically be supported with extensions to Azure TRE.
+
+The Azure Trusted Research Environment project is an accelerator to assist Microsoft customers and partners who want to build out Trusted Research environments on Azure. This project enables authorized users to deploy and configure secure workspaces and R&D tooling without a dependency on IT teams.  
 
 This project is typically implemented alongside a data platform that provides research ready datasets to TRE workspaces:
 
 ![Concepts](assets/TRE_Overview.png)
 
+Azure TRE has also been proven to handle unstructured data scenarios. For example, collaborative Computer Aided Design (CAD), with Product Lifecycle Management integration.
+
 TREs are not “one size fits all”, hence although the Azure TRE has a number of out of the box features, the project has been built be extensible, and hence tooling and data platform agnostic.
 
 Core features include:
 
 - Self-service for administrators – workspace creation and administration
-- Self-service for research teams – research tooling creation and administration
+- Self-service for R&D teams – R&D tooling creation and administration
 - Package and repository mirroring
 - Extensible architecture - build your own service templates as required
 - Microsoft Entra ID integration

templates/workspace_services/guacamole/user_resources/guacamole-azure-windowsvm/porter.yaml

@@ -1,7 +1,7 @@
 ---
 schemaVersion: 1.0.0
 name: tre-service-guacamole-windowsvm
-version: 1.2.4
+version: 1.2.6
 description: "An Azure TRE User Resource Template for Guacamole (Windows 10)"
 dockerfile: Dockerfile.tmpl
 registry: azuretre

templates/workspace_services/guacamole/user_resources/guacamole-azure-windowsvm/terraform/vm_config.ps1

@@ -44,15 +44,21 @@ $DaemonConfig = @"
 $DaemonConfig | Out-File -Encoding Ascii ( New-Item -Path $env:ProgramData\docker\config\daemon.json -Force )
 
 # R config
-# $RconfigFilePathWindows = C:\Progra~1\R\4.1.2\etc\Rprofile.site
-#Add-Content $RconfigFilePathWindows "local({`n    r <- getOption(`"repos`")`n    r[`"Nexus`"] <- `"${nexus_proxy_url}/repository/r-proxy/`"`n    options(repos = r)`n})"
-# echo "local({`n    r <- getOption(`"repos`")`n    r[`"Nexus`"] <- `"${nexus_proxy_url}/repository/r-proxy/`"`n    options(repos = r)`n})" > $RconfigFilePathWindows
 $RConfig = @"
 local({
     r <- getOption("repos")
     r["Nexus"] <- "${nexus_proxy_url}/repository/r-proxy/"
     options(repos = r)
 })
 "@
-$RConfig | Out-File -Encoding Ascii ( New-Item -Path $Env:ProgramFiles\R\R-4.1.2\etc\Rprofile.site -Force )
 
+$RBasePath = "$Env:ProgramFiles\R"
+
+if (Test-Path $RBasePath) {
+  $RVersions = Get-ChildItem -Path $RBasePath -Directory | Where-Object { $_.Name -like "R-*" }
+
+  foreach ($RVersion in $RVersions) {
+      $ConfigPath = Join-Path -Path $RVersion.FullName -ChildPath "etc\Rprofile.site"
+      $RConfig | Out-File -Encoding Ascii (New-Item -Path $ConfigPath -Force)
+  }
+}