diff --git a/docs/aws-integration/efs-setup.md b/docs/aws-integration/efs-setup.md new file mode 100644 index 0000000000..3c8b8b2724 --- /dev/null +++ b/docs/aws-integration/efs-setup.md @@ -0,0 +1,93 @@ +Sure! Here's the documentation for integrating AWS Elastic File System (EFS) for static code upload in CloudCV, formatted as a markdown file named `efs-setup.md`: + +```markdown +# Documentation: AWS Elastic File System (EFS) Integration for Static Code Upload in CloudCV + +## **Overview** +This documentation provides a comprehensive guide to integrating AWS Elastic File System (EFS) for the static code upload feature in CloudCV. EFS enables scalable and persistent storage to handle large-scale data uploads seamlessly. Follow the steps below to configure, mount, and use EFS within your environment. + +--- + +## **Prerequisites** +Ensure the following requirements are met before starting: +- An active AWS account. +- IAM permissions to create and manage EFS, EC2 instances, and security groups. +- Installed AWS CLI (v2 or higher). +- SSH access to the EC2 instance used for the upload pipeline. + +--- + +## **Step 1: Create an EFS File System** +1. **Log in** to the AWS Management Console and navigate to **Elastic File System (EFS)**. +2. Click **Create File System** and configure: + - **VPC**: Select the same VPC as your EC2 instance. + - **Performance Mode**: General Purpose is recommended for most use cases. + - **Throughput Mode**: Choose between Bursting or Provisioned based on workload. +3. Click **Create** to provision the EFS and note down its ID for later use. + +--- + +## **Step 2: Configure Security Groups** +1. **Update EFS Security Group**: + - Add an inbound rule to allow NFS traffic (port 2049) from the EC2 instance's security group. +2. **Update EC2 Security Group**: + - Add an outbound rule to allow NFS traffic to the EFS security group. + +--- + +## **Step 3: Mount EFS on EC2 Instance** +1. **Install the NFS client**: + ```bash + sudo apt update && sudo apt install -y nfs-common # For Ubuntu/Debian + sudo yum install -y nfs-utils # For Amazon Linux/CentOS + ``` +2. **Create a mount directory**: + ```bash + sudo mkdir -p /mnt/efs + ``` +3. **Mount the EFS file system**: + ```bash + sudo mount -t nfs4 -o nfsvers=4.1 fs-.efs..amazonaws.com:/ /mnt/efs + ``` +4. **Verify the mount**: + ```bash + df -h | grep efs + ``` + +--- + +## **Step 4: Automate Mounting** +To make the EFS mount persistent across reboots: +1. Edit the `/etc/fstab` file: + ```plaintext + fs-.efs..amazonaws.com:/ /mnt/efs nfs4 defaults,_netdev 0 0 + ``` +2. Test the configuration: + ```bash + sudo umount /mnt/efs + sudo mount -a + ``` + +--- + +## **Step 5: Upload Data to EFS** +Transfer files to the mounted EFS directory: +```bash +scp -i ec2-user@:/mnt/efs/ +``` + +--- + +## **Step 6: Integrate with CloudCV** +1. **Update Configuration**: + - Modify the `code-upload` pipeline’s configuration to use the mounted EFS directory as the upload target. + - Example: Set the upload path to `/mnt/efs` in `settings.py` or `config.yaml`. +2. **Test Integration**: + - Run a sample upload to confirm the integration is working correctly. + +--- + +## **Troubleshooting** +- **Permission Denied**: Verify the IAM roles, security group rules, and mount permissions. +- **Mount Failure**: Ensure the EC2 instance and EFS are in the same VPC. +- **Slow Transfers**: Optimize the throughput mode or consider enabling encryption. \ No newline at end of file diff --git a/evalai/EvalAI b/evalai/EvalAI new file mode 160000 index 0000000000..39e2f2e7d7 --- /dev/null +++ b/evalai/EvalAI @@ -0,0 +1 @@ +Subproject commit 39e2f2e7d74f391f04bcde3f4089ac4a51929813 diff --git a/frontend_v2/src/app/components/challenge/challengesubmit/challengesubmit.component.ts b/frontend_v2/src/app/components/challenge/challengesubmit/challengesubmit.component.ts index aefdd5ba8b..748384f0a0 100644 --- a/frontend_v2/src/app/components/challenge/challengesubmit/challengesubmit.component.ts +++ b/frontend_v2/src/app/components/challenge/challengesubmit/challengesubmit.component.ts @@ -529,8 +529,10 @@ export class ChallengesubmitComponent implements OnInit { SELF.submissionError = ''; if (SELF.components) { SELF.components['_results'].forEach((element) => { - element.value = ''; - element.message = ''; + if (element.label !== 'submissionOptions') { + element.value = ''; + element.message = ''; + } }); } } @@ -629,6 +631,7 @@ export class ChallengesubmitComponent implements OnInit { self.globalService.setFormValueForLabel(self.components, 'method_description', ''); self.globalService.setFormValueForLabel(self.components, 'project_url', ''); self.globalService.setFormValueForLabel(self.components, 'publication_url', ''); + self.isSubmissionUsingUrl = false; // Clear the submission type marker }); self.isSubmitted = true; } diff --git a/frontend_v2/src/app/components/nav/footer/footer.component.html b/frontend_v2/src/app/components/nav/footer/footer.component.html index 0aff778f6e..cb22d2f78f 100644 --- a/frontend_v2/src/app/components/nav/footer/footer.component.html +++ b/frontend_v2/src/app/components/nav/footer/footer.component.html @@ -12,7 +12,7 @@
EvalAI
  • diff --git a/frontend_v2/src/assets/images/twitter.png b/frontend_v2/src/assets/images/twitter.png index ba65e6350b..7a7e077368 100644 Binary files a/frontend_v2/src/assets/images/twitter.png and b/frontend_v2/src/assets/images/twitter.png differ diff --git a/frontend_v2/src/assets/images/twitter_black.png b/frontend_v2/src/assets/images/twitter_black.png index 02816933c7..b4034d2009 100644 Binary files a/frontend_v2/src/assets/images/twitter_black.png and b/frontend_v2/src/assets/images/twitter_black.png differ diff --git a/frontend_v2/src/assets/images/twitter_dark.png b/frontend_v2/src/assets/images/twitter_dark.png index 933466b33e..7ae2489a23 100644 Binary files a/frontend_v2/src/assets/images/twitter_dark.png and b/frontend_v2/src/assets/images/twitter_dark.png differ diff --git a/new b/new new file mode 100644 index 0000000000..e69de29bb2 diff --git a/pytorch_model.bin b/pytorch_model.bin new file mode 100644 index 0000000000..229c055ff4 --- /dev/null +++ b/pytorch_model.bin @@ -0,0 +1 @@ +Found. Redirecting to https://cdn-lfs.hf.co/bert-base-uncased/097417381d6c7230bd9e3557456d726de6e83245ec8b24f529f60198a67b203a?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27pytorch_model.bin%3B+filename%3D%22pytorch_model.bin%22%3B&response-content-type=application%2Foctet-stream&Expires=1736098505&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTczNjA5ODUwNX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy5oZi5jby9iZXJ0LWJhc2UtdW5jYXNlZC8wOTc0MTczODFkNmM3MjMwYmQ5ZTM1NTc0NTZkNzI2ZGU2ZTgzMjQ1ZWM4YjI0ZjUyOWY2MDE5OGE2N2IyMDNhP3Jlc3BvbnNlLWNvbnRlbnQtZGlzcG9zaXRpb249KiZyZXNwb25zZS1jb250ZW50LXR5cGU9KiJ9XX0_&Signature=HBiOo28EiP%7ES4CR9j8CE2jIaoHTvm82VGqguTfmBW2ZbvTPFfQbfBk4QoTYW4YkssHsatUzivSfC1p-Mf8GNG7XVktnVobGj75wlFo-qoLT5sBIbfxXRajiyMAYNvUkMyzjiTJJ-TPTLoEwqJqoBEBQi2RSskmZ5RShEKniNSOmjk77UKnXpVS7BIwtKLb17H0lIt%7Ehx2%7EpyURVT9Xaeh1Nzon9ANwbrkmx--7cIqJDK3xtfmnUaIfUZGHPTsA7lPwfr%7EKZC-rn6SEn-HaNBwkVG28-CAkJvVa0WbRb9Z3sfVNoNZL1Pp8K-Jffo4H0Bi70p4fqx7xpxqEoNtCaAjQ__&Key-Pair-Id=K3RPWS32NSSJCE \ No newline at end of file diff --git a/scripts/workers/code_upload_worker_utils/install_dependencies.sh b/scripts/workers/code_upload_worker_utils/install_dependencies.sh index 9e32bdd3e4..0ee3be54a2 100755 --- a/scripts/workers/code_upload_worker_utils/install_dependencies.sh +++ b/scripts/workers/code_upload_worker_utils/install_dependencies.sh @@ -1,6 +1,6 @@ #!/bin/bash -# installing AWS CLI +# Installing AWS CLI curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip ./aws/install @@ -15,7 +15,7 @@ echo "### AWS CLI Configured" curl -o aws-iam-authenticator https://amazon-eks.s3.us-west-2.amazonaws.com/1.17.7/2020-07-08/bin/linux/amd64/aws-iam-authenticator chmod +x ./aws-iam-authenticator mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin -echo 'export PATH=$PATH:$HOME/bin' >> ~/.bashrc +echo 'export PATH=$PATH/$HOME/bin' >> ~/.bashrc echo "### iam-authenticator Installed" # Configure kubeconfig @@ -38,6 +38,7 @@ kubectl create configmap fluent-bit-cluster-info \ --from-literal=read.head='On' \ --from-literal=read.tail='Off' \ --from-literal=logs.region=$AWS_DEFAULT_REGION -n amazon-cloudwatch + # Use FluentD compatible FluentBit insights kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit-compatible.yaml echo "### Container Insights Installed" @@ -49,19 +50,19 @@ cat /code/scripts/workers/code_upload_worker_utils/persistent_volume.yaml | sed kubectl apply -f /code/scripts/workers/code_upload_worker_utils/persistent_volume_claim.yaml kubectl apply -f /code/scripts/workers/code_upload_worker_utils/persistent_volume_storage_class.yaml -# Install cilium +# Install Cilium # Cilium is being used to provide networking and network policy kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.9/install/kubernetes/quick-install.yaml echo "### Cilium Installed" sleep 120s; -# Apply cilium network policy -# echo "### Setting up Cilium Network Policy..." -# cat /code/scripts/workers/code_upload_worker_utils/network_policies.yaml | sed "s/{{EVALAI_DNS}}/$EVALAI_DNS/" | kubectl apply -f - -# echo "### Cilium EvalAI Network Policy Installed" +# Apply Cilium Network Policy +echo "### Setting up Cilium Network Policy..." +cat /code/scripts/workers/code_upload_worker_utils/network_policies.yaml | sed "s/{{EVALAI_DNS}}/$EVALAI_DNS/" | kubectl apply -f - +echo "### Cilium EvalAI Network Policy Installed" -# Set ssl-certificate +# Set SSL certificate echo $CERTIFICATE | base64 --decode > scripts/workers/certificate.crt # Running Submission Worker diff --git a/scripts/workers/code_upload_worker_utils/network_policies.yaml b/scripts/workers/code_upload_worker_utils/network_policies.yaml index 2732aa92fe..cbeeaf40c0 100644 --- a/scripts/workers/code_upload_worker_utils/network_policies.yaml +++ b/scripts/workers/code_upload_worker_utils/network_policies.yaml @@ -10,6 +10,7 @@ spec: - toFQDNs: - matchPattern: {{EVALAI_DNS}} - matchPattern: "*.ubuntu.com" + - matchPattern: "*.example.com" # Add any other necessary domains here - toEndpoints: - matchLabels: "k8s:io.kubernetes.pod.namespace": kube-system @@ -21,3 +22,9 @@ spec: rules: dns: - matchPattern: "*" + - toPorts: + - ports: + - port: "443" + protocol: TCP + - port: "80" + protocol: TCP \ No newline at end of file