Securing large language models with a reverse proxy

In a previous post, I explained how to host a private ChatGPT using Docker and Traefik. I didn’t spend a lot of time on the security aspect of the project.

I see many people asking how to expose their large language model on Internet and ask how to secure it. Since most (all?) open-source projects have adopted the OpenAI API, it uses standard HTTP. Therefore you can use all the traditional techniques to secure your large language model with a reverse proxy.

Since a reverse proxy sits between the clients and the server, there are 2 connections to secure:

1. Between the clients and the reverse proxy
2. Between the reverse proxy and the server

There are 2 obvious goals you may want to achieve when securing those connections:

1. Eavesdroppers can’t read your data
2. Only allowed people can use your service

The first one is achieved through encryption, the second one with authentication.

The obvious connect you would want to secure is the client-facing one, as it will be exposed out there. The second connection might stay on the same server, on the same internal network, or it might not. Only you know and ultimately, it’s your choice and responsibility.

Encryption

HTTPS was invented 30 years ago according to Wikipedia, and since the emergence of Let’s Encrypt, SSL certificates are free for everyone and are renewed automatically thanks to the ACME protocol. There’s really no reason to not use one.

You’ll find guides to configure HTTPS on your reverse proxy of choice. Some of them (Traefik for example) even handle the ACME protocol themselves, while others (Apache httpd and Nginx) rely on an external script (certbot).

Authentication

While you don’t have many options for encryption, you have several for authentication:

• basic authentication
• mutual TLS
• IP-based authentication
• API key
• JWT

The choice depends on what you are trying to achieve and how much effort you want to put into it.

IP-based auth is completely transparent to the client, the IP source is allowed or it’s not. The configuration is done on the server side and all of them support a kind of IP whitelist. It’s called Access control on Apache httpd, access module with nginx and the IP Allow List middleware in Traefik.

Basic authentication is almost compatible with everything (one notable exception is `Ollama client`). Usually libraries such as requests directly support it. Worst case scenario: you have to compute the base64 and set the header yourself. Look at mod_auth_basic for httpd, auth basic module for nginx and the Basic Auth middleware with Traefik.

Mutual TLS is when the client needs to provide a certificate from a known-source to the server to be allowed to connect. That’s a bit more annoying to manage as you have to hand certificates to your clients. However, this is way more robust than basic auth since it’s not feasible to brute-force it. Look at VerifyClient of mod_ssl for httpd, ssl_verify_client on nginx and TLS clientAuth setting in Traefik.

API key is like basic authentication except you choose the name of the header. You generate a secure API key (usually a very long alphanumerical string), you give it to the client. When the client connects, it sets a header such as “X-APIKEY” with the key and your server must ensure the key is valid. You can kind of hack it with Access control on httpd. You can use map and if statements with nginx. There is no native way to accomplish this in Traefik but you can use plugins such as this one (disclaimer: I have not tested it).

Finally, JWT a bit like an API key except the user has to authenticate with the server though another channel first. Then the user gets a token for the app. The token is placed in a standard header and the server can verify the token is valid. As far as I know, httpd does not support it natively yet, although there is mod_autht_jwt for httpd 2.5. Nginx supports it with auth_jwt module and again, you must use plugins with Traefik.

Example with Traefik

Here’s a sample Traefik config to implement both IP-based and basic authentications. Replace myservice with the name of your service, the IP addresses with yours and the username:password with a valid one.

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.myservice.rule=Host(`myservice.example.com`)"
      - "traefik.http.routers.myservice.tls=true"
      - "traefik.http.routers.myservice.entrypoints=websecure"
      - "traefik.http.routers.myservice.tls.certresolver=le"
      - "traefik.http.routers.myservice.middlewares=myservice-ipwhitelist, myservice-auth"
      - "traefik.http.routers.myservice.service=myservice"
      - "traefik.http.services.myservice.loadbalancer.server.port=80"
      - "traefik.http.services.myservice.loadbalancer.server.scheme=http"
      - "traefik.http.middlewares.myservice-ipwhitelist.ipwhitelist.sourcerange=10.0.0.0/8, 192.168.69.69"
      - "traefik.http.middlewares.myservice-auth.basicauth.users=myuser:my-password"

You can generate a valid username:password pair with the following command:

echo $(htpasswd -nB user) | sed -e s/\\$/\\$\\$/g

Have fun and stay safe!