Python script called in a .NET API in docker, dependencies not found

Question:

Hello I have a project with the following folder structure (abstracted):

.
└── root/
    ├── ExRate_Service/
    │   ├── MyOwnPythonScripts... 
    │   └── Program.py
    ├── ExRate_API/
    │   ├── ExRate_API/
    │   │   ├── Classes...
    │   │   ├── ExRate_API.csproj
    │   │   └── Program.cs
    │   ├── ExRate_API.Tests/
    │   │   └── Tests...
    │   └── ExRate_API.sln
    └── Dockerfile

The Dockerfile contains the following and builds without error:

FROM python:3.9.10-slim-buster AS build
WORKDIR /app
RUN pip install numpy pandas requests python-dotenv tensorflow scikit-learn keras matplotlib
COPY ./ExRate_Service .
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS package
WORKDIR /app
COPY ExRate_API .
RUN dotnet publish -c Release -o out
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS final
WORKDIR /app
COPY --from=build /app ExRate_Service
COPY --from=build /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=package /app/out ExRate_API
ENTRYPOINT ["dotnet", "ExRate_API/ExRate_API.dll"]
EXPOSE 80

Here is the code from the API which calls the python script (I have a second version of this method which runs if it is not in a container, which works fine when the API is run locally without a container)

        public string getOutputInContainer(string targetCurrency, string baseCurrency)
        {
            _logger.LogInformation($"Container method running");

            var scriptPath = "/app/ExRate_Service/Program.py";

            var output = string.Empty;
            var process = new Process
            {
                StartInfo = new ProcessStartInfo
                {
                    FileName = "/usr/local/lib/python3.9",
                    Arguments = scriptPath + $" -b {targetCurrency} -t {baseCurrency}",
                    UseShellExecute = false,
                    RedirectStandardOutput = true,
                    RedirectStandardError = true,
                    CreateNoWindow = true
                },
                EnableRaisingEvents = true
            };

            process.ErrorDataReceived += (sender, e) =>
            {
                if (!string.IsNullOrEmpty(e.Data))
                {
                    _logger.LogInformation($"Error from process: {e.Data}");
                }
            };
            process.OutputDataReceived += (sender, e) =>
            {
                if (!string.IsNullOrEmpty(e.Data))
                {
                    _logger.LogInformation($"Output from process: {e.Data}");
                    output += e.Data;
                }
            };

            process.Start();
            process.BeginErrorReadLine();
            process.BeginOutputReadLine();
            process.WaitForExit();

            _logger.LogInformation($"Raw output: {output}");

            var historicalData = JsonConvert.DeserializeObject<Dictionary<string, object>>(output.Substring(0, output.IndexOf("}") + 1));
            var forecast = JsonConvert.DeserializeObject<Dictionary<string, object>>(output.Substring(output.IndexOf("}") + 1));

            _logger.LogInformation($"Forecast: {JsonConvert.SerializeObject(forecast)}");

            var result = new Dictionary<string, Dictionary<string, object>>();
            result.Add("historicalData", historicalData ?? new Dictionary<string, object>());
            result.Add("forecast", forecast ?? new Dictionary<string, object>());

            _logger.LogInformation($"Result: {JsonConvert.SerializeObject(result)}");

            var json = JsonConvert.SerializeObject(result, Formatting.Indented);

            _logger.LogInformation($"Final JSON: {json}");

            return json;
        }

When the API is running using this command docker run -ti --rm -p 8080:80 exrate in the container i get the following error:

info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app/
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/1.1 GET http://localhost:8080/api/GetExRateForecast/USD&EUR - -
warn: Microsoft.AspNetCore.HttpsPolicy.HttpsRedirectionMiddleware[3]
      Failed to determine the https port for redirect.
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[0]
      Executing endpoint 'ExRate_API.Controllers.GetExRateForecastController.Get (ExRate_API)'
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[3]
      Route matched with {action = "Get", controller = "GetExRateForecast"}. Executing controller action with signature Microsoft.AspNetCore.Mvc.IActionResult Get(System.String, System.String) on controller ExRate_API.Controllers.GetExRateForecastController (ExRate_API).
info: ExRate_API.Controllers.GetExRateForecastController[0]
      Request received for baseCurrency: USD, targetCurrency: EUR
info: ExRate_API.Controllers.GetExRateForecastController[0]
      Container method running
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
      Executed action ExRate_API.Controllers.GetExRateForecastController.Get (ExRate_API) in 35.3081ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
      Executed endpoint 'ExRate_API.Controllers.GetExRateForecastController.Get (ExRate_API)'
fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HMOTGC8R42M3", Request id "0HMOTGC8R42M3:00000002": An unhandled exception was thrown by the application.
      System.ComponentModel.Win32Exception (0x80004005): The FileName property should not be a directory unless UseShellExecute is set.         at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
         at System.Diagnostics.Process.Start()
         at ExRate_API.DataFromService.GetExRateForecast.getOutputInContainer(String targetCurrency, String baseCurrency) in /app/ExRate_API/DataFromService/GetExRateForecast.cs:line 97
         at ExRate_API.Controllers.GetExRateForecastController.Get(String baseCurrency, String targetCurrency) in /app/ExRate_API/Controllers/GetExRateForecastController.cs:line 30
         at lambda_method2(Closure , Object , Object[] )
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/1.1 GET http://localhost:8080/api/GetExRateForecast/USD&EUR - - - 500 0 - 155.0379ms

when I go into the bash console of the container, i can see python3 is installed as well as the dependeinces, they are all there. so why can’t my script find numpy (assuming thats the one which fails first) or any other?

I am not a docker expert and this is my first time trying to build a container of this complexity.

The full project is hosted on github here

Any help or pointers would be appreicated.

Asked By: David Beesley

||

Answers:

Okay have sorted the above issue.

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS package
WORKDIR /app

COPY ExRate_API .
RUN dotnet publish -c Release -o out


FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS final
WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends 
    python3.9 
    python3-pip 
    && 
    apt-get clean && 
    rm -rf /var/lib/apt/lists/*

RUN pip install numpy pandas requests python-dotenv tensorflow scikit-learn keras matplotlib

COPY ./ExRate_Service ExRate_Service
COPY --from=package /app/out ExRate_API

ENTRYPOINT ["dotnet", "ExRate_API/ExRate_API.dll"]
EXPOSE 80
Answered By: David Beesley
Categories: questions Tags: , , , ,
Answers are sorted by their score. The answer accepted by the question owner as the best is marked with
at the top-right corner.