slow start up script in rodan-main with a lot of data

[homework36]

copied from #1241

I figured out that it is because we have too many files already and therefore this line takes a very very long time to run in scripts/start and scripts/start-celery as this

Rodan/scripts/start

Line 11 in 245db99

chmod -R a+rwx /rodan

Currently we first run this on the NFS host

chmod -R a+rwx /var/lib/docker/volumes/rodan_resources/_data

and then verify using

find /var/lib/docker/volumes/rodan_resources/_data ! -perm 0777 -exec ls -ld {} \;

And then we replace the original command section in production.yml with tail -f /dev/null.
After we launch the docker stack, we manually go into the container and comment out the line chmod -R a+rwx /rodan in /run/start or /run/start-celery and then execute.

docker ps --filter "name=rodan_rodan-main" --format "{{.ID}}" | xargs -I {} docker exec {} sed -n "11p" /run/start
docker ps --filter "name=rodan_rodan-main" --format "{{.ID}}" | xargs -I {} bash -c 'docker exec {} sed -i "11s/^/ #/" /run/start && docker exec {} /run/start'


docker ps --filter "name=rodan_celery" --format "{{.ID}}" | xargs -I {} docker exec {} sed -n "8p" /run/start-celery
docker ps --filter "name=rodan_celery" --format "{{.ID}}" | xargs -I {} bash -c 'docker exec {} sed -i "8s/^/ #/" /run/start-celery && docker exec {} /run/start-celery'

docker ps --filter "name=rodan_gpu-celery" --format "{{.ID}}" | xargs -I {} docker exec {} sed -n "8p" /run/start-celery
docker ps --filter "name=rodan_gpu-celery" --format "{{.ID}}" | xargs -I {} bash -c 'docker exec {} sed -i "8s/^/ #/" /run/start-celery && docker exec {} /run/start-celery'

docker ps --filter "name=rodan_py3-celery" --format "{{.ID}}" | xargs -I {} docker exec {} sed -n "8p" /run/start-celery
docker ps --filter "name=rodan_py3-celery" --format "{{.ID}}" | xargs -I {} bash -c 'docker exec {} sed -i "8s/^/ #/" /run/start-celery && docker exec {} /run/start-celery'

We might need to rebuild all the images to remove the line. But we also need to make sure the permission status... Still thinking about the solution.

[homework36]

Temporary solution:

We build docker images under tag fix_startscript which has commented out those two lines for production server.
We need to further discuss possible stable solutions for the future.