docker-compose up is stuck on attaching to
Question:
I have written several python scripts that will backtest trading strategies. I am attempting to deploy these through docker compose.
The feeder container copies test files to a working directory where the backtester containers will pick them up and process them. The processed test files are then sent to a "completed work" folder. Some CSV files that the backtester outputs are then written to an NFS share on another computer. I should mention that this is all running on Ubuntu 20.04.
As far as I can tell everything should be working, but for some reason the "docker-compose up" command hangs up on "Attaching to". There should be further output with print statements (I’ve unbuffered the Dockerfiles so those should show up). I’ve also left it running for a while to see if anything was getting processed and it looks like the containers never started up. I’ve looked at all the other threads dealing with this and have not found a solution that has worked to resolve this.
Any insight is very very appreciated. Thanks.
Here is the docker-compose file:
version: '3.4'
services:
feeder:
image: feeder
build:
context: .
dockerfile: ./Dockerfile.feeder
volumes:
- /home/danny/io:/var/lib/io
worker1:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker1:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
worker2:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker2:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
worker3:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker3:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
Here is the Dockerfile for the backtester:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
# RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
# USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "backtester5.py"]
Here is the Dockerfile for the feeder:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
# RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
# USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "feeder.py"]
Here is the last message that shows up:
Attaching to project4_feeder_1, project4_worker1_1, project4_worker2_1, project4_worker3_1
Answers:
It’s been three weeks with no responses, but I just wanted to update with what I’ve found. In all cases where I’ve left "docker-compose up" running it eventually started.
At times it took 30 minutes, but it started every time.
I faced the same problem and fix it with this tip:
resolved It turns out if I run my docker command with "python3 -u" it will force python to run unbuffered. It was a buffering issue.
I have written several python scripts that will backtest trading strategies. I am attempting to deploy these through docker compose.
The feeder container copies test files to a working directory where the backtester containers will pick them up and process them. The processed test files are then sent to a "completed work" folder. Some CSV files that the backtester outputs are then written to an NFS share on another computer. I should mention that this is all running on Ubuntu 20.04.
As far as I can tell everything should be working, but for some reason the "docker-compose up" command hangs up on "Attaching to". There should be further output with print statements (I’ve unbuffered the Dockerfiles so those should show up). I’ve also left it running for a while to see if anything was getting processed and it looks like the containers never started up. I’ve looked at all the other threads dealing with this and have not found a solution that has worked to resolve this.
Any insight is very very appreciated. Thanks.
Here is the docker-compose file:
version: '3.4'
services:
feeder:
image: feeder
build:
context: .
dockerfile: ./Dockerfile.feeder
volumes:
- /home/danny/io:/var/lib/io
worker1:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker1:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
worker2:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker2:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
worker3:
image: backtester
build:
context: .
dockerfile: ./Dockerfile
volumes:
- /home/danny/io/input/workers/worker3:/var/lib/io/input
- /home/danny/io/input/completedwork:/var/lib/io/archive
- /nfs/tests:/var/lib/tests
Here is the Dockerfile for the backtester:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
# RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
# USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "backtester5.py"]
Here is the Dockerfile for the feeder:
# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.8-slim-buster
# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
# Creates a non-root user with an explicit UID and adds permission to access the /app folder
# For more info, please refer to https://aka.ms/vscode-docker-python-configure-containers
# RUN adduser -u 5678 --disabled-password --gecos "" appuser && chown -R appuser /app
# USER appuser
# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "feeder.py"]
Here is the last message that shows up:
Attaching to project4_feeder_1, project4_worker1_1, project4_worker2_1, project4_worker3_1
It’s been three weeks with no responses, but I just wanted to update with what I’ve found. In all cases where I’ve left "docker-compose up" running it eventually started.
At times it took 30 minutes, but it started every time.
I faced the same problem and fix it with this tip:
resolved It turns out if I run my docker command with "python3 -u" it will force python to run unbuffered. It was a buffering issue.