I had been thinking using Wordpress for a personal blog is kind of wasteful for a long long time. After all, I don’t have dynamic content, it’s really just a bunch of text. I wanted to migrate to a static-file CMS for a long time but I never had the courage to do so. I recently had a few days of downtime, and so finally I did it. I decided to use Hugo as it was the most popular option at the time....
It has been a long time since I wanted to control connections from/to Docker containers but I could never find a simple enough solutions. We can control reverse proxy settings (Traefik) using labels but we can’t apply iptables rules with them? Nonsense. If you add to this that every container lives in a namespace, and namespaces can have their iptables rules, you have there an easy solution. So I wrote a Bash script that listen to Docker events....
In a previous post, I explained how to host a private ChatGPT using Docker and Traefik. I didn’t spend a lot of time on the security aspect of the project. I see many people asking how to expose their large language model on Internet and ask how to secure it. Since most (all?) open-source projects have adopted the OpenAI API, it uses standard HTTP. Therefore you can use all the traditional techniques to secure your large language model with a reverse proxy....
There was a recent dramatic improvement on the speed of LLM’s on CPU thanks to llamafile’s author. She goes on extensively about it on her blog but the short version is: expect 7-billion parameters to be usable on consumer-grade CPU even in Q8. Now it’s certainly possible to self-host a coding assistant with llamafile, continue.dev and Docker on a VPS. Let’s see how to achieve that. I’ll use Docker + Traefik but you can easily convert it to anything else (native + nginx for example)....
Reading Ollama discord channel, I notice many people want to self-host their chatGPT with Docker and don’t know how to do it. Here’s how to host the whole stack with docker compose. Here’s my docker-compose.yml including the mitmproxy from the previous article. version: "3" services: ollama: build: ollama user: 1001:1001 environment: - OLLAMA_HOST=0.0.0.0 - OLLAMA_DEBUG=1 - OLLAMA_KEEP_ALIVE=60m volumes: - /etc/localtime:/etc/localtime:ro - ollama_models:/home/ollama/.ollama/models mitmproxy: image: mitmproxy/mitmproxy command: mitmweb --web-host 0.0.0.0 --web-port 8080 --mode reverse:http://ollama:11434@11434 --verbose --anticache --anticomp depends_on: - ollama labels: - "traefik....
Sometimes you connect a new tool to one of your servers and it doesn’t work as expected. You are sure you follow the documentation or tutorials but you don’t get the expected results. Before you throw away everything, you should check what’s actually going on between the 2 applications. And if none of them supports logging requests and responses, you can use mitmproxy for troubleshooting. As the name imply (MITM = Man In the Middle), mitmproxy sits between both applications and intercepts all the traffic....
Ollama I have recently started to use Ollama and I was unimpressed by some models as they did not follow instructions, especially in their output format. I knew about model system prompt but I thought it was fixed in the model. Then I found out you could change the system prompt at run time with the /set system command and immediately, most models responded as expected. That was so much better!...
I have just bought a mini PC based on Intel N100 CPU. Initially, I was going to buy another Raspberry PI or a used “TinyMiniMicro” PC, but I decided to have a look at the current mini PC offering. I am glad I did. On a major Chinese reseller website, I saw a lot of similar products with the Intel N100 CPU so I had a look at reviews (here here) and boy, this thing is powerful (for its size)....
By default, resources available to containers are not limited. However, sometimes, you want to make sure a container is not going to use too much processing power or memory. To achieve such a thing, in the docker-compose.yml file, add the following sections to the service you want to restrict: deploy: resources: limits: cpus: "1.0" memory: 100M memswap_limit: 100M This will effectively limit the container to use at most one CPU and 100 megabytes of memory....
If you cannot upgrade your OpenSSH client and/or server to fix CVE-2023-48795, also known as the Terrapin attack, the way to mitigate it is to disable the vulnerable ciphers as Red Hat explains very well. If you have a recent OpenSSH version, you can disable the the ciphers by adding “-” before them in the Ciphers and MACs options. This works for both the ssh client config (/etc/ssh/ssh_config by default) and the ssh server config (/etc/ssh/sshd_config)....