A few months back, I decided to convert an existing Dotnet core application to run on Linux as a proof of concept. As part of this work, I decided to run it inside a Docker container, which meant I could have my application running on Linux without worrying about setting up all of the dependencies on the server like Dotnet runtime, Apache, etc.
Now I fully realise that Windows containers exist and that I could have tried to get them to work. But the aim for me wasn’t to run on Docker, it was to run on Linux. Docker just came in handy because it meant I didn’t need to install all dependencies onto my Linux server – I could just fire up Docker.
Now, much like communism, Docker is pretty great--in theory. Once the process begins, however, you quickly realise just how many worms are inside the can you’ve just opened.
But unlike communism, in the end, this worked! Here’s the output from my web application showing some server details. Notice any differences?
So with that in mind, here's a list of issues that you should probably know about.
Table Of Contents
- Building the docker image
- Image Layers and caching
- Getting the app to start up
- Support for third party libraries
- Windows Authentication
- Runtime Errors
- System fonts
- Unspecified Culture
- HTTPS and Certificates
- Network Share Paths
- Case sensitive URLs
- Other Things
- Disk Space
- The Aftermath
Building the Docker image
Image Layers and Caching
The first step was to package my application up inside a Docker image. Microsoft has some great articles on how to do this, but they are the usual only-works-for-a-simple-demo examples. My application is a bit larger and has multiple project files--hardly novel and hardly new. Fortunately, a programmer named Tometchy has a great blog post on how to set this up. After following his advice, I created my new Dockerfile and ran:
docker build src/myproject/Dockerfile
... and it built! However... it was taking forever!
Unless you structure your Dockerfile in a very particular way, your NuGet package restores will always take forever to restore. Instead of using a local package cache, Dotnet will do a complete NuGet restore for every build. This means builds will take minutes instead of the usual seconds. So instead of doing this:
COPY . ./ Dotnet build You can take advantage of Docker’s layer caching like so: COPY src/myproject/myproject.csproj COPY nuget.config ./nuget.config RUN dotnet restore src/myproject/myproject.csproj \ --configfile nuget.config \ --packages packages COPY . ./ RUN dotnet build
This means that every new Docker build won’t do a full NuGet restore—that will only happen when you change your project file. This meant I could build my images in a few seconds. Now that it was working, it was time to run my application.
Getting the app to start up
When running an app inside Docker, Localhost suddenly isn't localhost. The network has been virtualised! Localhost doesn't mean what you think it means, it’s now referring to the Docker container itself, not the host. There is a simple fix for this - use host.docker.internal instead of localhost. This will work when developing locally, but when running on a server you’re probably better off specifying the full hostname and hope that DNS is set up correctly. Once I changed this, at least my application could connect to my Seq instance to start logging startup errors.
Support for third party libraries
Once I fixed my references to localhost, the next problem that I had was not all of my DLL’s were loading properly. At startup I would get this error:
For PDF management I’m using a library named AbcPdf. It’s a great library with heaps of functionality that I’ve been using for years. There’s just one problem. AbcPdf isn't supported on Linux. There are just no Linux binaries, full stop. Sure, this was hidden inside their documentation somewhere, but the only way to find this out was to actually launch my application and see if it crashed at startup.
This was pretty confusing, because the code worked great on Windows. (I particularly like the stack trace with unreadable characters). But after a bit of googling on their website, I found this gem:
“Windows only – not Linux or Xamarin”.
This was a problem. My only options were:
- Investigate other PDF libraries that have Linux support
- Refactor out all of the AbcPdf code and run it on Windows
I chose Option 2. I refactored out all of the code to a separate application that I would run on Windows and slapped an API wrapper around it. This was a bit of work, but there wasn’t that much code there, and what was there could be pulled out pretty easily. After a day, I ended up with a nice API for creating and merging PDFs:
This was also quite nice as I knew there was another application that needed some PDF functionality, and now I had a nice and reusable API to call.
The next thing that I needed to do was to check that all of my other dependencies had Linux support. So here’s the full list of libraries that I’m using:
- Hangfire - long running tasks
- Serilog + Seq - logging
- Swashbuckle - swagger API
- AWS SDK - talking to S3
- EPPlus - excel file manipulation
Fortunately, all of them have fantastic Linux support. Except for the one that didn’t.
The next problem was happening at startup:
System.InvalidOperationException: No authenticationScheme was specified, and there was no DefaultChallengeScheme found. The default schemes can be set using either AddAuthentication(string defaultScheme) or AddAuthentication(Action
This didn’t make any sense. It’s the same code from Windows… but the more I looked at it, I realised that this was probably the culprit:
First off, the app isn’t using IIS, so the chances are, IIS authentication probably isn’t going to work. So after much googling, I found that while it is possible to get windows authentication working on Linux, to set it up you need to have access to a domain controller – something that I expect no developer would have--or should have! This is clearly not a practical solution. My solution was to move Windows auth out of the Linux server and move it to a different application (a topic for another post). But, to get you started, I found out I could have the application use IIS auth for when running under Windows, and a different auth for when running under Linux:
This is kind of cool, but really it probably should be a config that is set at startup, not determined by the code at runtime depending on the OS that it’s running on.
My application was now loading! I could click around and it was responding. It barely worked, but hey, it was loading. Now to start fixing all of the runtime errors.
Inside my application I had a little bit of code that created a list of all of the system fonts installed on the server:
When this code ran, I got this error:
Not the most helpful error message I've ever seen, but needless to say, managing font files just doesn't work. I managed to find this github issue that had a few workarounds, but I never got around to trying them. I ended up moving the same code to the Pdf Api that I mentioned earlier.
This is the kind of thing that ended up being quite common when testing my application on Linux. Something causes the app to crash that normally works on Windows, you google it to find a couple of possible workarounds and try out a few.
Notice anything weird going on here?
Ever seen the ¤ character before? Neither had I, but apparently it's the currency sign used to denote an unspecified currency. This means you need to set the culture inside your code instead of relying on the server’s culture:
Problem solved. That was a good one!
HTTPS and Certificates
Linux and Windows handle certificate trusting differently. One of the API’s that I’m calling had a certificate that used older ciphers. Apparently, in windows this was fine, but inside Linux, this was not fine and the SSL connection was rejected. I ended up having to modify the runtime openssl configuration inside the Docker container:
RUN sed 's/[email protected]=2/[email protected]=1/' \ /etc/ssl/openssl.cnf > /etc/ssl/openssl.cnf.changed && \ sed 's/TLSv1.2/TLSv1.1/' /etc/ssl/openssl.cnf.changed \ /etc/ssl/openssl.cnf.changed2 && \ mv /etc/ssl/openssl.cnf.changed2 /etc/ssl/openssl.cnf
Yikes. Isn't it great how docker simplifies things?
When you make a call to fetch a list of Timezones, the list that is returned is completely different when running on Linux compared to Windows! Apparently this is because the list of time zones is returned by the OS... and of course the two do it completely differently. There is a library to work around this, but boy is it a doozy to catch during runtime.
Network Share Paths
Writing to a Windows network share is suddenly much more difficult. My solution was to mount the network share on the host using samba (via smbclient), and then mapping this mounted share to a Docker volume, and then reading from that. It then finally works, after only adding two more levels of abstraction! But… can you change the permissions on the newly created file? Let’s try it:
Oh well, I guess not.
After a while you kind of get used to seeing messages that effectively say "this stuff works on Windows but not on Linux".
Case sensitive URLs
My URLs are suddenly case sensitive! As my app is running under Kestral + Linux, not Windows + IIS, one of those two has determined that a URL like MyFile.js is suddenly incorrect, when it should technically be myfile.js. This wasn’t too much of a problem, but it caught me out a few times.Other Things
There were a few other things that I needed to keep in mind.
When building Docker images, I would constantly run out of disk space. It’s really annoying and it's quite difficult to troubleshoot andwork out where all of the space is going and how to clean it up. Yes, you have the Docker prune command, but that wasn’t really working. I ended up having to impose some pretty strict “only use this much space” policies to my hyper-v image, which were really difficult to get right.
If you're not careful, the Docker images can become huge. One of the third-party libraries that I was using was putting 100 MBs of stuff in my application's bin directory. This meant every -Docker image push+pull would be using a lot more bandwidth than I expected. Easily done and difficult to spot.
My CICD pipeline suddenly becomes very different. Instead of using the normal templated solution that I have for all of my other applications I suddenly need to have a custom build for Docker. The deploys are also completely different - better, MUCH better, but oh so different. Not something that I really thought through, but something that needs to be considered.
You need to keep passwords and all other secrets out of your image. This means passing through all secrets as environment variables and making sure your application is loading them in at runtime. Nothing extraordinary here, but some more work that needed to be done to make sure it was working, and setting up an env.list file.
So after all of those woes, I managed to get my application fully up and running:
I added a new page to dump out some runtime information, and it now displays the following:
This is pretty cool! This opens up a world of possibilities:
- My application runs faster. Dotnet on Linux performs better than on Windows. It uses less memory and is able to handle more traffic
- Deploys are vastly improved:
- They are much simpler - just pull down the Docker image and run it
- A lot faster - practically zero downtime as it only takes a second to startup
- Instant roll-back – running a previous version only takes a second
- Scaling out my application is much easier – just run the Docker image on another server
- My app can run easily on Azure or AWS
- My Docker image can be scanned for out-of-date dependencies or security problems
- Integration tests are now much easier to run.I can create a Docker compose file that sets up the entire environment (database, API’s, etc.) and runs my tests
- I can upgrade to a newer version of the Dotnet runtime without having to worry about what’s installed on the server
- I can run my app on a Kubernetes cluster While moving to Docker wasn’t exactly a walk in the park, now that I have that knowledge moving my other applications won’t be nearly as difficult. Now the question is which one to tackle next!