Prod pipeline v5 #112

hellais · 2024-11-11T16:03:36Z

Devops related work to get OONI Pipeline v5 production ready

* Make use of clickhouse idealista role * Refactor nftables role * Tighten up ssh users related configuration * Improve the bootstrap host playbook

github-actions · 2024-11-11T16:04:24Z

Terraform Run Output 🤖

Format and Style 🖌`failure`

Initialization ⚙️`success`

Validation 🤖`success`

Validation Output


$ terraform validate
Success! The configuration is valid.

Plan 📖`success`

Plan: 0 to add, 1 to change, 0 to destroy.

Show Plan


$ terraform plan
module.adm_iam_roles.tls_private_key.oonidevops: Refreshing state... [id=b49a9fdb9f720320340226016efe24808dd68203]
module.ansible_inventory.local_file.ansible_inventory: Refreshing state... [id=b6de844ed8d384f890fa6f467502390de843f758]
random_id.artifact_id: Refreshing state... [id=8Ujqew]
random_password.prometheus_metrics_password: Refreshing state... [id=none]
random_password.jwt_secret: Refreshing state... [id=none]
module.ansible_inventory.null_resource.ansible_update_known_hosts: Refreshing state... [id=236461505953331670]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooniapi_user.aws_ses_email_identity.ooniapi: Refreshing state... [[email protected]]
module.adm_iam_roles.aws_key_pair.oonidevops: Refreshing state... [id=oonidevops]
module.ooniapi_ooniauth.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniauth]
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Reading...
module.oonidevops_github_user.aws_iam_user.oonidevops_github: Refreshing state... [id=oonidevops-github]
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooni_backendproxy.data.aws_ssm_parameter.ubuntu_22_ami: Reading...
module.ooniapi_oonirun_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonirun]
module.ooniapi_cluster.aws_cloudwatch_log_group.ooniapi_services: Refreshing state... [id=ooni-ecs-group/ooniapi-ecs-cluster]
module.adm_iam_roles.data.aws_iam_policy_document.assume_role: Read complete after 0s [id=4022892340]
module.ooniapi_ooniprobe.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role]
module.adm_iam_roles.aws_secretsmanager_secret.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe]
module.ooniapi_oonifindings.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role]
data.aws_availability_zones.available: Reading...
aws_secretsmanager_secret.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ]
module.ooniapi_cluster.aws_iam_role.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role]
module.ooniapi_ooniauth.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-ooniauth-td:61/ooniapi-service-ooniauth]
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
module.ooniapi_oonirun.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-oonirun-td:56/ooniapi-service-oonirun]
module.ooniapi_user.aws_iam_user.ooniapi: Refreshing state... [id=oonidevops-ooniapi]
module.ooni_backendproxy.data.aws_ssm_parameter.ubuntu_22_ami: Read complete after 0s [id=/aws/service/canonical/ubuntu/server/22.04/stable/current/amd64/hvm/ebs-gp2/ami-id]
module.ooniapi_oonirun.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role]
aws_acm_certificate.ooniapi_frontend: Refreshing state... [id=arn:aws:acm:eu-central-1:905418398257:certificate/190205f1-392d-425c-a059-7006ca8c8c46]
module.ooniapi_user.aws_secretsmanager_secret.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx]
module.ooniapi_ooniprobe.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-ooniprobe]
module.adm_iam_roles.aws_iam_policy.oonidevops: Refreshing state... [id=arn:aws:iam::905418398257:policy/OONIDevopsPolicy]
module.oonidevops_github_user.aws_secretsmanager_secret.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd]
module.ooniapi_oonifindings.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-oonifindings-td:14/ooniapi-service-oonifindings]
module.ooniapi_oonifindings_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-oonifindings]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
aws_s3_bucket.ooniapi_codepipeline_bucket: Refreshing state... [id=codepipeline-ooniapi-eu-central-1-f148ea7b]
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Reading...
aws_s3_bucket.oonith_codepipeline_bucket: Refreshing state... [id=codepipeline-oonith-eu-central-1-f148ea7b]
aws_secretsmanager_secret.jwt_secret: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/jwt_secret-NUESvS]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_ooniprobe.data.aws_ecs_container_definition.ooniapi_service_current[0]: Read complete after 0s [id=arn:aws:ecs:eu-central-1:905418398257:task-definition/ooniapi-service-ooniprobe-td:39/ooniapi-service-ooniprobe]
module.ooniapi_oonirun.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonirun]
module.oonidevops_github_user.aws_iam_policy.oonidevops_github: Refreshing state... [id=arn:aws:iam::905418398257:policy/oonidevops-github-policy]
module.ooniapi_oonifindings_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Reading...
aws_secretsmanager_secret.prometheus_metrics_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniprobe]
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Reading...
module.ooniapi_oonirun_deployer.data.aws_caller_identity.current: Read complete after 1s [id=905418398257]
module.ooniapi_user.aws_secretsmanager_secret.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr]
module.ooniapi_ooniauth.aws_iam_role.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codepipeline: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codepipeline-ooniapi-ooniauth]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Reading...
module.ooniapi_cluster.data.aws_ssm_parameter.ecs_optimized_ami: Read complete after 1s [id=/aws/service/ecs/optimized-ami/amazon-linux-2/recommended]
module.ooniapi_oonifindings.aws_cloudwatch_log_group.ooniapi_service: Refreshing state... [id=ooni-ecs-group/ooniapi-service-oonifindings]
module.oonidevops_github_user.aws_iam_access_key.oonidevops_github: Refreshing state... [id=AKIA5FTZELIYXDN55SMS]
module.ooniapi_ooniauth_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_oonirun_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonirun]
module.ooniapi_ooniprobe.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniprobe-task-role:ooniapi-service-ooniprobe-task-role]
module.ooniapi_oonifindings.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonifindings-task-role:ooniapi-service-oonifindings-task-role]
module.ooniapi_user.aws_iam_access_key.ooniapi: Refreshing state... [id=AKIA5FTZELIYSK2XEVOT]
module.ooniapi_user.aws_iam_user_policy.ooniapi: Refreshing state... [id=oonidevops-ooniapi:oonidevops-ooniapi-policy]
module.ooniapi_cluster.aws_iam_instance_profile.container_host: Refreshing state... [id=ooniapi-ecs-cluster]
module.ooniapi_cluster.aws_iam_role_policy.container_host: Refreshing state... [id=ooniapi-ecs-cluster-container-host-role:ooniapi-ecs-cluster-instance-role-policy]
module.ooniapi_cluster.aws_ecs_cluster.main: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:cluster/ooniapi-ecs-cluster]
module.ooniapi_ooniprobe_deployer.data.aws_caller_identity.current: Read complete after 0s [id=905418398257]
module.ooniapi_oonirun.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-oonirun-task-role:ooniapi-service-oonirun-task-role]
module.adm_iam_roles.aws_iam_role.oonidevops: Refreshing state... [id=oonidevops]
module.ooniapi_oonifindings_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-oonifindings]
module.adm_iam_roles.aws_secretsmanager_secret_version.oonidevops_deploy_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|terraform-20240925140131946100000002]
module.oonidevops_github_user.aws_iam_user_policy_attachment.oonidevops_github: Refreshing state... [id=oonidevops-github-20240313195612421500000001]
aws_secretsmanager_secret_version.jwt_secret: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/jwt_secret-NUESvS|terraform-20240310182536838400000005]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniprobe]
aws_secretsmanager_secret_version.prometheus_metrics_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw|terraform-20240314200140936700000008]
data.aws_secretsmanager_secret_version.prometheus_metrics_password: Reading...
module.ooniapi_ooniauth_deployer.aws_iam_role.codepipeline: Refreshing state... [id=codepipeline-ooniapi-ooniauth]
module.oonidevops_github_user.aws_secretsmanager_secret_version.oonidevops_github: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/github_user/access_key_json-9JTJgd|terraform-20240519071250187000000004]
aws_route53_record.ooniapi_frontend_cert_validation["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__a064be8aa084a037ff9fa5e3e541c87d.ooniprobe.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__ef17825e5fd9713f596344bdd9626f5e.8.th.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__48cd4e71cee9930614228176b7deefb9.ooniauth.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__05c891caeb4509d4cd7f9c24d8b6dbd0.oonirun.dev.ooni.io._CNAME]
aws_route53_record.ooniapi_frontend_cert_validation["api.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL__cd4729fc0c282e771d056e719a7bdf4f.api.dev.ooni.io._CNAME]
data.aws_secretsmanager_secret_version.prometheus_metrics_password: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni_services/prometheus_metrics_password-M8BbRw|AWSCURRENT]
module.ooniapi_ooniauth.aws_iam_role_policy.ooniapi_service_task: Refreshing state... [id=ooniapi-service-ooniauth-task-role:ooniapi-service-ooniauth-task-role]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_secret_access_key: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_secret_access_key-L0DQDr|terraform-20240314200140914600000006]
module.ooniapi_user.aws_secretsmanager_secret_version.aws_access_key_id: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooniapi_user/aws_access_key_id-EcXOBx|terraform-20240314200140918400000007]
aws_codestarconnections_connection.oonidevops: Refreshing state... [id=arn:aws:codestar-connections:eu-central-1:905418398257:connection/6bd492f6-c11d-43ec-92b0-24c47700d528]
data.aws_secretsmanager_secret_version.deploy_key: Reading...
module.oonipg.random_password.pg_password: Refreshing state... [id=none]
module.terraform_state_backend.data.aws_region.current: Reading...
module.terraform_state_backend.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.network.aws_vpc.main: Refreshing state... [id=vpc-0e382f3ad89286de9]
module.oonipg.aws_secretsmanager_secret.pg_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/pg_password-OjzOJC]
aws_acm_certificate_validation.ooniapi_frontend: Refreshing state... [id=0001-01-01 00:00:00 +0000 UTC]
module.terraform_state_backend.aws_s3_bucket.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.bucket_policy[0]: Read complete after 0s [id=2666303363]
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Reading...
module.terraform_state_backend.data.aws_iam_policy_document.aggregated_policy[0]: Read complete after 0s [id=2666303363]
module.terraform_state_backend.aws_dynamodb_table.with_server_side_encryption[0]: Refreshing state... [id=oonidevops-dev-terraform-state-lock]
data.aws_secretsmanager_secret_version.deploy_key: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/deploy_key-2ebqSe|AWSCURRENT]
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Reading...
module.ooni_th_droplet.data.cloudinit_config.ooni_th_docker: Read complete after 0s [id=1719060339]
module.ooni_th_droplet.digitalocean_droplet.ooni_th_docker[0]: Refreshing state... [id=451534580]
module.oonipg.aws_secretsmanager_secret_version.pg_password: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/pg_password-OjzOJC|terraform-20240310155428358300000002]
module.ooniapi_ooniauth_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniauth-eu-central-1]
module.ooniapi_ooniprobe_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-ooniprobe-eu-central-1]
module.ooniapi_oonifindings_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonifindings-eu-central-1]
module.ooniapi_oonirun_deployer.aws_iam_policy.codebuild: Refreshing state... [id=arn:aws:iam::905418398257:policy/service-role/codebuild-oonirun-eu-central-1]
module.ooniapi_oonifindings_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonifindings]
module.ooniapi_oonirun_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-oonirun]
module.ooniapi_ooniauth_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniauth]
module.ooniapi_ooniprobe_deployer.aws_iam_role.codebuild: Refreshing state... [id=codebuild-ooniapi-ooniprobe]
module.ooniapi_ooniauth_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniauth]
module.ooniapi_oonirun_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonirun]
module.ooniapi_oonifindings_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-oonifindings]
module.ooniapi_ooniprobe_deployer.aws_codebuild_project.ooniapi: Refreshing state... [id=arn:aws:codebuild:eu-central-1:905418398257:project/ooniapi-ooniprobe]
module.network.aws_internet_gateway.gw: Refreshing state... [id=igw-0c080e9b235ed29d1]
module.ooni_backendproxy.aws_alb_target_group.oonibackend_proxy: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/oobpx20240422160053180600000001/1ec17e88e2467a5b]
module.ooniapi_oonirun.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OrunM-20240917211808753100000003/93d4b2b6dc76acac]
module.ooniapi_ooniprobe.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OproM-20240917211808753100000004/1f2ea1732205872c]
module.ooniapi_cluster.aws_security_group.web: Refreshing state... [id=sg-0187eedfe39538357]
module.ooniapi_oonifindings.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OfinM-20240917211808752800000002/48adcfb18ae34d30]
module.oonipg.aws_security_group.pg: Refreshing state... [id=sg-005ca579eb9c08cda]
module.ooniapi_ooniauth.aws_alb_target_group.ooniapi_service: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:targetgroup/OautM-20240917211808752700000001/91bcad756924a3a7]
module.network.aws_route_table.private: Refreshing state... [id=rtb-011463437da96c77b]
module.network.aws_subnet.private[1]: Refreshing state... [id=subnet-0b899a7ad10406d06]
module.network.aws_subnet.private[0]: Refreshing state... [id=subnet-09314a43ec89d6331]
module.network.aws_subnet.public[0]: Refreshing state... [id=subnet-0e7a4478be988463f]
module.network.aws_subnet.public[1]: Refreshing state... [id=subnet-0b18966cccfc9d5ef]
module.network.aws_route_table.public: Refreshing state... [id=rtb-0ccb0852e6a365a95]
module.ooniapi_cluster.aws_security_group.container_host: Refreshing state... [id=sg-0aa6a97400b619de3]
module.terraform_state_backend.aws_s3_bucket_versioning.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_public_access_block.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.terraform_state_backend.aws_s3_bucket_server_side_encryption_configuration.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.network.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-0e7933e6b804ff2c1]
module.network.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-0c9cc0f117ef15fe7]
module.network.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-0dbd7fb16801ee049]
module.network.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-08ab18165bf481054]
module.terraform_state_backend.aws_s3_bucket_policy.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.oonipg.aws_db_subnet_group.pg: Refreshing state... [id=ooni-tier0-postgres-dbsng]
module.ooni_backendproxy.aws_security_group.nginx_sg: Refreshing state... [id=sg-050fe64c8d862ab5e]
module.ooniapi_frontend.aws_alb.ooniapi: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:loadbalancer/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6]
module.ooniapi_cluster.aws_launch_template.container_host: Refreshing state... [id=lt-0e328a8671f870c64]
module.terraform_state_backend.time_sleep.wait_for_aws_s3_bucket_settings[0]: Refreshing state... [id=2024-03-10T15:06:17Z]
module.terraform_state_backend.aws_s3_bucket_ownership_controls.default[0]: Refreshing state... [id=oonidevops-dev-terraform-state]
module.ooniapi_cluster.aws_autoscaling_group.container_host: Refreshing state... [id=ooniapi-ecs-cluster20240310192644083800000003]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_https: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd]
module.ooniapi_frontend.aws_alb_listener.ooniapi_listener_http: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/d9b2448464179cd1]
aws_route53_record.ooniapi_frontend_main: Refreshing state... [id=Z055356431RGCLK3JXZDL_api.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["oonirun.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_oonirun.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["ooniprobe.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniprobe.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["8.th.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_8.th.dev.ooni.io_A]
aws_route53_record.ooniapi_frontend_alt["ooniauth.dev.ooni.io"]: Refreshing state... [id=Z055356431RGCLK3JXZDL_ooniauth.dev.ooni.io_A]
module.oonipg.aws_db_instance.pg: Refreshing state... [id=db-27N7Q6XIBNASFCOXN4N7C762L4]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/5a872da4cc6b9135]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonifindings_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/cdc4e8e8eabb56f2]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/65e6f5e3aca0a4e5]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/cefeff0d8aa3118a]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/2b09ed268181ba4f]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_oonirun_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/551c4128bb282fa4]
module.ooniapi_frontend.aws_alb_listener_rule.ooniapi_th: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/01ee7503374be8ff]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniauth_rule: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/128c53ea760208fc]
module.ooniapi_frontend.aws_lb_listener_rule.ooniapi_ooniprobe_rule_host: Refreshing state... [id=arn:aws:elasticloadbalancing:eu-central-1:905418398257:listener-rule/app/ooni-tier0-api-frontend/52df1e7ac0eb1ea6/2f500e01e10ba5cd/b436b91883ae7c86]
aws_secretsmanager_secret_version.oonipg_url: Refreshing state... [id=arn:aws:secretsmanager:eu-central-1:905418398257:secret:oonidevops/ooni-tier0-postgres/postgresql_url-w62CTZ|terraform-20240310182536837800000004]
aws_route53_record.postgres_dns: Refreshing state... [id=Z091407123AEJO90Z3H6D_postgres.dev.ooni.nu_CNAME]
module.ooniapi_ooniprobe.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniprobe-td]
module.ooniapi_oonifindings.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonifindings-td]
module.ooniapi_ooniauth.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-ooniauth-td]
module.ooniapi_oonirun.aws_ecs_task_definition.ooniapi_service: Refreshing state... [id=ooniapi-service-oonirun-td]
module.ooniapi_ooniprobe.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-ooniprobe]
module.ooniapi_oonifindings.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-oonifindings]
module.ooniapi_ooniauth.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-ooniauth]
module.ooniapi_oonirun.aws_ecs_service.ooniapi_service: Refreshing state... [id=arn:aws:ecs:eu-central-1:905418398257:service/ooniapi-ecs-cluster/ooniapi-service-oonirun]
module.ooniapi_ooniprobe_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-ooniprobe]
module.ooniapi_oonifindings_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-oonifindings]
module.ooniapi_ooniauth_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-ooniauth]
module.ooniapi_oonirun_deployer.aws_codepipeline.ooniapi: Refreshing state... [id=ooniapi-oonirun]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform planned the following actions, but then encountered a problem:

  # module.ooniapi_oonifindings_deployer.aws_codepipeline.ooniapi will be updated in-place
  ~ resource "aws_codepipeline" "ooniapi" {
        id             = "ooniapi-oonifindings"
        name           = "ooniapi-oonifindings"
        tags           = {}
        # (5 unchanged attributes hidden)

      - trigger {
          - provider_type = "CodeStarSourceConnection" -> null

          - git_configuration {
              - source_action_name = "Source" -> null

              - push {
                  - branches {
                      - excludes = [] -> null
                      - includes = [
                          - "master",
                        ] -> null
                    }
                }
            }
        }

        # (4 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.


Pusher	@hellais
Action	pull_request
Environment	dev
Workflow	.github/workflows/check_terraform.yml
Last updated	Thu, 14 Nov 2024 17:16:48 GMT

github-actions · 2024-11-11T16:05:10Z

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖`success`

Show Execution


$ ansible-playbook playbook.yml --check --diff -i ../tf/modules/ansible_inventory/inventories/inventory-dev.ini
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: Could not match supplied host pattern, ignoring: monitoring.ooni.org
[WARNING]: Could not match supplied host pattern, ignoring: openvpn-
server1.ooni.io
[WARNING]: Could not match supplied host pattern, ignoring: data1.htz-
fsn.prod.ooni.nu
[WARNING]: Could not match supplied host pattern, ignoring: data3.htz-
fsn.prod.ooni.nu
[WARNING]: Could not match supplied host pattern, ignoring: notebook.ooni.org

PLAY [Ensure all hosts are bootstrapped correctly] *****************************
skipping: no hosts matched

PLAY [Update monitoring config] ************************************************
skipping: no hosts matched

PLAY [Setup OpenVPN server] ****************************************************
skipping: no hosts matched

PLAY [Deploy oonidata clickhouse hosts] ****************************************
skipping: no hosts matched

PLAY [Deploy oonidata worker nodes] ********************************************
skipping: no hosts matched

PLAY [Deploy notebook host] ****************************************************
skipping: no hosts matched

PLAY RECAP *********************************************************************


Pusher	@hellais
Action	pull_request
Working Directory
Workflow	.github/workflows/check_ansible.yml
Last updated	Thu, 14 Nov 2024 17:17:13 GMT

…ipeline-v5 * 'prod-pipeline-v5' of github.com:ooni/devops: Add inter-server comms port to nft rules

hellais added 2 commits November 11, 2024 14:24

Retrieve digital ocean secrets from AWS

0f35fa0

Work towards production deployment of data pipeline v5

c79c006

* Make use of clickhouse idealista role * Refactor nftables role * Tighten up ssh users related configuration * Improve the bootstrap host playbook

hellais added 14 commits November 11, 2024 17:09

Move clickhouse related functionality into specific role

8b33a52

Update to the clickhouse multi-host config

a8929b2

Fixes to the variables of clickhouse config

a7f86fb

Add tailnet setup

d2ea36e

Run clickhouse keeper on notebook host

87acee4

Debugging click cluster setup

12fee2b

Add documentation on how to setup MergeTree replication

bfcab27

Add inter-server comms port to nft rules

e47abde

Add more docs

53e1b4e

Merge branch 'prod-pipeline-v5' of github.com:ooni/devops into prod-p…

e76a6d7

…ipeline-v5 * 'prod-pipeline-v5' of github.com:ooni/devops: Add inter-server comms port to nft rules

Add chrony, journald and resolved config

2d412a2

Move test of resolved into handlers

cadc7b4

Deploy oonidata worker on data1.htz-fsn.prod.ooni.nu

e1de316

Become miniconda user before running conda commands

4a6714c

hellais mentioned this pull request Nov 15, 2024

Setup replication on production clickhouse instance #111

Closed

hellais merged commit db8334a into main Nov 22, 2024
3 checks passed

hellais deleted the prod-pipeline-v5 branch November 22, 2024 15:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prod pipeline v5 #112

Prod pipeline v5 #112

hellais commented Nov 11, 2024 •

edited

Loading

github-actions bot commented Nov 11, 2024 •

edited

Loading

github-actions bot commented Nov 11, 2024 •

edited

Loading

Prod pipeline v5 #112

Prod pipeline v5 #112

Conversation

hellais commented Nov 11, 2024 • edited Loading

github-actions bot commented Nov 11, 2024 • edited Loading

Terraform Run Output 🤖

Format and Style 🖌failure

Initialization ⚙️success

Validation 🤖success

Plan 📖success

github-actions bot commented Nov 11, 2024 • edited Loading

Ansible Run Output 🤖

Ansible Playbook Recap 🔍

Ansible playbook output 📖success

hellais commented Nov 11, 2024 •

edited

Loading

github-actions bot commented Nov 11, 2024 •

edited

Loading

Format and Style 🖌`failure`

Initialization ⚙️`success`

Validation 🤖`success`

Plan 📖`success`

github-actions bot commented Nov 11, 2024 •

edited

Loading

Ansible playbook output 📖`success`