Fast FreeBSD with ZFS on Vultr 29 May 2025

For the past year I've been running several Postgres databases in FreeBSD VPS's hosted in Vultr. I run Postgres in jails, FreeBSD's abstraction for OS-level virtualization. Jails work very well when paired with ZFS, so I prefer to run my hosts using ZFS for the root file system.

Unfortunately, although Vultr provides FreeBSD images, those images use UFS on root instead of ZFS. Because of that, I've been setting up my VPS's by creating them from a custom bootonly ISO, and then installing FreeBSD from scratch. This has worked relatively well, except for the fact that for some reason the boot process would pause for about 30-60 seconds. When creating a VPS from Vultr's FreeBSD images, the boot was fast.

This made me very concerned about rebooting my hosts. No more! Today I discovered the problem.

tldr; Create your VPS from the Vultr-provided FreeBSD image, then attach a bootonly ISO and reinstall the OS from scatch.

𐡷

FreeBSD is fantastic. The tooling is not as polished as what I was used to when operating Illumos systems, but it seems to me (from my very naive point of view) that the community is larger. It's missing a sophisticated RBAC system and SMF, but it has ZFS on root, OS-level virtualization, and a good network stack. For purposes I've needed in the past year, being quick to set up and having a lot of people contributing to keeping the package system up to date was more important than being perfect.

The entire reason for me to maintain my own VPS's was to run the latest version of Postgres, 17.5 at the time of writing, and to run database clusters as inexpensively as possible. My experience using hosted Postgres is that providers lag anywhere from from 6 to 24 months behind major release cycles... when I started this migration Postgres 17.0 had just been released, but one of the major hosted Postgres providers I evaluated was still locked at Postgres 15!

Ultra slow boot times cause problems, though. The slower the boot time, the greater the downtime when upgrading db hosts. The greater the potential downtime, the more likely I am to delay non-critical updates—especially during high traffic periods for our applications.

This was something that has been nagging at me for months, so recently as I've been reviewing our system automation and cleaning up some deadline-driven compromises, I decided to finally find a solution to the problem. If the Vultr-provided images could boot fast, then somehow my custom install could be made fast.

𐡷

Having spun up two new (very small) VPS's, one with a custom install, the other from Vultr's FreeBSD 14 image, I went to work trying to find differences. Lo and behold, there were a lot.

First of all, I found a number of kernel modules and tunings provided by the the Vultr image.

The kernel modules aesni, crypto, and cc_htcp were loaded in /boot/loader.conf.

Various sysctls were configured in /etc/sysctl.conf:

hw.mds_disable=3
kern.ipc.maxsockbuf=67108864
net.inet.tcp.sendbuf_max=67108864
net.inet.tcp.recvbuf_max=67108864
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.cc.algorithm=htcp

While these were good things to discover, configuring my custom host with the same modules and values did not solve the problem; it would have been surprising if they had, to be honest. I was starting to believe that there might be some problem inherent to running ZFS in a VPS host in Vultr...

Then I did a closer comparison of the boot logs. I'd looked at /var/log/messages before, but never compared the values side by side.

Slow boot:

Hyper-V Version: 10.0.14393 [SP0]
Features=0x27e<TMREFCNT,SYNIC,SYNTM,APIC,HYPERCALL,VPINDEX,REFTSC>
PM Features=0x0 [C0]
Features3=0x8<PCPUDPE>
Timecounter "Hyper-V" frequency 10000000 Hz quality 2000
CPU: Intel Core Processor (Broadwell, no TSX, IBRS) (2394.29-MHz K8-class CPU)
  Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID>
  Structured Extended Features3=0x84000000<IBPB,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "Microsoft Hv"

Fast boot:

CPU: Intel Core Processor (Broadwell, no TSX, IBRS) (2394.52-MHz K8-class CPU)
  Origin="GenuineIntel" Id=0x306d2 Family=0x6 Model=0x3d Stepping=2
  Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x7a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID>
  Structured Extended Features3=0x84000000<IBPB,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
Hypervisor: Origin = "KVMKVMKVM"

Well, there's the problem! Why is my VPS running under Hyper-V, when it could be running under KVM???

After a short back-and-forth with Vultr support, I received the suggestion that I should create the host from Vultr's FreeBSD image, then attach my ISO and re-install the operating system. Doing this flags the host as FreeBSD in their system; they apply different configuration on the back end when they know that the host is a BSD.

After following this advice on a new host, I not only found that reboots were fast, but that the boot logs indicated the host was running under KVM.

...now to rebuild all my hosts.

𐡷

2025