FreeBSD - hpaluch/hpaluch.github.io GitHub Wiki

FreeBSD notes

My first encounter with FreeBSD was around '95 with version 2.x. Originally I happily used Linux distribution MCC Interim 1.0+ with kernel 1.0.4 on AMD 386SX machine with 4MB of RAM and 40MB IDE IBM disk.

However Linux kernel 1.2.x introduced serious issues:

  • it crashed on 1st access on Floppy driver, because of DMA overrun (kernel 1.0.x worked without problems on same HW). This was many years later found and fixed.
  • Linux console started to be terribly slow - especially when using colors compared to kernel 1.0.x

FreeBSD has always pretty fast console with fully working scrollback (using Scroll-Lock), while Linux used ugly hack (that prevent scrollback after flipping to another console and back) and even that hack was removed entirely recently. However I later returned to Linux because it was simply not possible to maintain system by downloading sources and rebuilding everything from scratch (in those times the only option to keep FreeBSD up to date).

So many years later (frustrated with systemd, snapcrap and flatpack future) I decided it to try it again using (FreeBSD-13.1-RELEASE-amd64-disc1.iso) on old MSI with NVidia chipset MCP55. However I'm simply not lucky - I was immediately hit with random files corruptions - even just reboot was enough to corrupt some additional files. It seems to be known issue with FreeBSD and NVidia MCP55 chipset as can be seen on:

Good news!

I have found that this MCP55 chipset works on FreeBSD 13.2 without problem in one of 2 conditions:

  1. With very old SATA-1 PCI SiI3112 controller
  2. in safe-mode (Boot menu Options -> Safe Mode: On)

In the past my old SATA-1 Maxtor driver worked without data corruption even on MCP55 chipset - however when I made big installation (all 3 BSDs across whole driver) - the corruption occured even on Maxtor + MCP55...

I'm now exploring what options in Safe mode are required. Important: FreeBSD 13.2 no longer use Forth menu (/boot/*.4th) but rather LUA in /boot/lua/*.lua. Here is simple way how I verified that LUA is used by this patch:

--- /boot/lua/menu.lua.orig	2023-04-07 06:24:33.000000000 +0200
+++ /boot/lua/menu.lua	2023-05-18 18:20:33.738859000 +0200
@@ -170,7 +170,7 @@
 		},
 		{
 			entry_type = core.MENU_SEPARATOR,
-			name = "Boot Options:",
+			name = "Boot Options (HP):",
 		},
 		-- acpi
 		{

I have verified that upon reboot that submenu changed. Now I have to study /boot/lua/core.lua:

-- /boot/lua/core.lua
local default_safe_mode = false

function core.setSafeMode(safe_mode)
	if safe_mode == nil then
		safe_mode = not core.sm
	end
	if safe_mode then
		loader.setenv("kern.smp.disabled", "1")
		loader.setenv("hw.ata.ata_dma", "0")
		loader.setenv("hw.ata.atapi_dma", "0")
		loader.setenv("hw.ata.wc", "0")
		loader.setenv("hw.eisa_slots", "0")
		loader.setenv("kern.eventtimer.periodic", "1")
		loader.setenv("kern.geom.part.check_integrity", "0")
	else
		loader.unsetenv("kern.smp.disabled")
        -- omited ...
	end
	core.sm = safe_mode
end

Doing bisection this line is important (as expected):

loader.setenv("hw.ata.ata_dma", "0")

Tried lower DMA mode (based on https://forums.freebsd.org/threads/how-to-add-command-to-boot.49556/) using:

-- loader.setenv("hw.ata.ata_dma", "0")
loader.setenv("hint.ata.5.mode", "UDMA33")

  • NOTE: You can find valid DMA modes in ata_str2mode() function on: https://cgit.freebsd.org/src/tree/sys/dev/ata/ata-all.c#n692 (as of May 2023)

    // sys/dev/ata/ata-all.c
static int
ata_str2mode(const char *str)
{

	if (!strcasecmp(str, "PIO0")) return (ATA_PIO0);
	if (!strcasecmp(str, "PIO1")) return (ATA_PIO1);
	if (!strcasecmp(str, "PIO2")) return (ATA_PIO2);
	if (!strcasecmp(str, "PIO3")) return (ATA_PIO3);
	if (!strcasecmp(str, "PIO4")) return (ATA_PIO4);
	if (!strcasecmp(str, "WDMA0")) return (ATA_WDMA0);
	if (!strcasecmp(str, "WDMA1")) return (ATA_WDMA1);
	if (!strcasecmp(str, "WDMA2")) return (ATA_WDMA2);
	if (!strcasecmp(str, "UDMA0")) return (ATA_UDMA0);
	if (!strcasecmp(str, "UDMA16")) return (ATA_UDMA0);
	if (!strcasecmp(str, "UDMA1")) return (ATA_UDMA1);
	if (!strcasecmp(str, "UDMA25")) return (ATA_UDMA1);
	if (!strcasecmp(str, "UDMA2")) return (ATA_UDMA2);
	if (!strcasecmp(str, "UDMA33")) return (ATA_UDMA2);
	if (!strcasecmp(str, "UDMA3")) return (ATA_UDMA3);
	if (!strcasecmp(str, "UDMA44")) return (ATA_UDMA3);
	if (!strcasecmp(str, "UDMA4")) return (ATA_UDMA4);
	if (!strcasecmp(str, "UDMA66")) return (ATA_UDMA4);
	if (!strcasecmp(str, "UDMA5")) return (ATA_UDMA5);
	if (!strcasecmp(str, "UDMA100")) return (ATA_UDMA5);
	if (!strcasecmp(str, "UDMA6")) return (ATA_UDMA6);
	if (!strcasecmp(str, "UDMA133")) return (ATA_UDMA6);
	return (-1);
}

  • also did not work:

    • UDMA0, UDMA2, UDMA44
    • non-smp mode
    • ACPI mode

The camcontrol negotiate ada0 showed: UDMA2

  • unfortunately corruption occured even at this transfer speed.

This patch that forces SATA-1 speed (similar to my NetBSD hack for 1TB Seagate freeze) significantly reduces (but not eliminates) data corruption:

--- /root/ata-sata.c	2023-05-20 17:53:12.287459000 +0200
+++ /usr/src/sys/dev/ata/ata-sata.c	2023-05-20 20:18:11.613931000 +0200
@@ -154,14 +154,25 @@
 int
 ata_sata_phy_reset(device_t dev, int port, int quick)
 {
+    const int PATCH_DEV_UNIT=5;
     struct ata_channel *ch = device_get_softc(dev);
-    int loop, retry, sata_rev;
+    int loop, retry, sata_rev,dev_unit=0;
     uint32_t val, val1;
 
+    dev_unit = device_get_unit(dev);
     sata_rev = ch->user[port < 0 ? 0 : port].revision;
+
+    if (dev_unit == PATCH_DEV_UNIT){
+	    sata_rev = 1; // FORCE Liks RESET to SATA-1 SPEED
+            device_printf(dev, "HPv6-PATCH:%d devunit='%d': FORCING SATA-1 sata_rev=0x%x quick=%d, port=%d\n",
+			    __LINE__,dev_unit,sata_rev, quick, port);
+    }
+
     if (sata_rev > 0)
 	quick = 0;
 
+    device_printf(dev, "HPv6:%d devunit='%d' sata_rev=0x%x quick=%d, port=%d\n",__LINE__,dev_unit,sata_rev, quick, port);
+
     if (quick) {
 	if (ata_sata_scr_read(ch, port, ATA_SCONTROL, &val))
 	    return (0);

Nope it does not help...

However this one - I just copied all DMA quirks from ata-siiliconimage.c to ata-nvidia.c seems to do the trick:

--- /root/ata-nvidia.c	2023-05-21 16:55:02.982165000 +0200
+++ /usr/src/sys/dev/ata/chipsets/ata-nvidia.c	2023-05-21 17:00:16.699407000 +0200
@@ -266,6 +266,10 @@
     ch->hw.status = ata_nvidia_status;
     ch->flags |= ATA_NO_SLAVE;
     ch->flags |= ATA_SATA;
+
+    device_printf(ch->dev, "HPv9: %s:%d activating DMA quirks\n",__FILE__,__LINE__);
+    ch->dma.boundary = 8192;
+    ch->dma.segsize = 15 * DEV_BSIZE;
     return 0;
 }

I need to do more rigorous testing to come to conclusion...

Workaround: Using PCI SATA controller

Now confirmed - when I attach disk to this antique PCI SATA controller - then FreeBSD can be installed without any issue:

atapci4: <SiI 3112 SATA150 controller>
  port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f
  mem 0xfcfffc00-0xfcfffdff
  irq 16 at device 1.0 on pci1

No longer FS corruption observed.

However found that this SiI controller will simply freeze after Identify phase, when connected to new Seagate 1TB disk - see NetBSD page for details.

How to detect FS corruption

In my case it started simply with crashing some programs (Signal 11) on boot including SSHd (which is a bit unfortunate. On 2nd installation there was corrupted /usr/lib/libarchive.so.7 so I was unable to work with any archives - however it worked to copy this file from installation DVD to system. The broken libarchive caused SIGBUS instead. Also there was corrupted /usr/share/misc/termcap.db so I quickly learned that it can be recreated with cap_mkdb termcap.

You can run this command to verify most files in installed BSD:

freebsd-update IDS

If your system has just few corrupted files, you can mount installation DVD and copy those files directly from DVD (there is live filesystem and fortunately it seems to have same binaries as installed system). Also copy of few essential binaries should be under /rescue folder of your system.

If your system is still enough usable you can even copy or extract somewhere installation *.txz archives from DVD's '/usr/freebsd-dist/` folder

FreeBSD firewall

Unlike other BSD systems (where pf is used), FreeBSD uses ipfw as default - see https://docs.freebsd.org/en/books/handbook/firewalls/

Here is simple snippet, that I use in /etc/rc.conf to allow only SSH access from single IP address:

# https://docs.freebsd.org/en/books/handbook/firewalls/
firewall_enable="YES"
firewall_type="workstation"
firewall_myservices="22/tcp"
# replace "192.168.0.X" with your allowed client address
firewall_allowservices="192.168.0.X"
# see /var/log/security for firewall log
firewall_logdeny=YES

To see current rules you can try ipfw list

To see what is happening under hood and how are rules constructed please look directly into /etc/rc.firewall script.

Downloading sources

To download sources using git, see https://docs.freebsd.org/en/books/handbook/mirrors/#git Please note, that git will transfer around 1.5GB of files (tested on Nov 2022)

Example disk layout

I have installed FreeBSD 13.1 under Proxmox VE 7.2-11 (KVM) using:

  • CPU: 1x kvm64 (not recommended but good for portability - if you plan to use same VM on different servers.
  • 2GB RAM
  • VirtioBlk disk, 16GB
  • VirtioNet network card

Here is partitioning as seen by fdisk:

fdisk vtbd0

******* Working on device /dev/vtbd0 *******
parameters extracted from in-core disklabel are:
cylinders=33288 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=33288 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
    start 64, size 33554368 (16383 Meg), flag 80 (active)
	beg: cyl 0/ head 1/ sector 2;
	end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

And disklabel with root filesystem and swap:

disklabel vtbd0s1
# /dev/vtbd0s1:
8 partitions:
#          size     offset    fstype   [fsize bsize bps/cpg]
  a:   29360128          0    4.2BSD        0     0     0
  b:    4194240   29360128      swap                    
  c:   33554368          0    unused        0     0     # "raw" part, don't edit

How to see listening sockets

In FreeBSD only sockstat command provides information which process is listening on which port (netstat can show sockets but not processes. For example:

$ sockstat -l -P tcp,udp

USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sshd       832   3  tcp6   *:22                  *:*
root     sshd       832   4  tcp4   *:22

NOTE: Other BSD systems use fstat to show both open files and sockets including listening.

Additionally to see important network resources (mbufs - buffers for networking) use:

$ netstat -m

257/1528/1785 mbufs in use (current/cache/total)
256/760/1016/506387 mbuf clusters in use (current/cache/total/max)
256/760 mbuf+clusters out of packet secondary zone in use (current/cache)
...

If all mbufs are in use you will be in trouble, for example:

To quote: https://rerepi.wordpress.com/2008/04/19/tuning-freebsd-sysoev-rit/

When there’s no any free mbuf clusters available FreeBSD enters the zonelimit state and stops to answer to any network requests. You can see it as the zoneli state in the output of the top command.

If you run busy network service you will find interesting information of Listen Queue with command - from http://nginx.org/en/docs/freebsd_tuning.html:

$ netstat -Lan

Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen                           Local Address
tcp4  0/0/128                          *.22
tcp6  0/0/128                          *.22
unix  0/0/4                            /var/run/devd.pipe
unix  0/0/4                            /var/run/devd.seqpacket.pipe

According to man netstat these 3 numbers have following meaning:

  1. Number of unaccepted connections (qlen)
  2. amount of unaccepted incomplete connections (incqlen)
  3. maximum number of queued connections (maxqlen)

Another command; To see established connections in real-time you can run systat -netstat

How to see swap usage

Use:

$ swapinfo -h

Device              Size     Used    Avail Capacity
/dev/ada0p2         8.0G       0B     8.0G     0%

FreeBSD console notes

Here is list of my favourite packages that I install on all FreeBSD systems (tested 13.2 RELEASE):

pkg install curl doas lynx mc smartmontools tmux vim wget

To allow all users in group wheel to run doas (like sudo) without password I create /usr/local/etc/doas.conf with contents:

permit nopass :wheel

Keep you base system up-to-date using:

freebsd-update fetch

If you suspect that some core binary is corrupted you can verify checksums using:

script ~/ids.log
freebsd-update IDS
exit # quit logging to ~/ids.log by exiting redirected shell process

Nice full-screen overview of system load:

systat -vmstat

Notice that FreeBSD version of systat requires dash (-) before vmstat "display". Also use :q followed by ENTER (like Vi(m)) to quit systat.

For top I use top -aP command where

  • -a will show command arguments
  • -P will show per/cpu statistics.

Enable PS/2 mouse on console

If you have real great PS/2 mouse you can enable it in /etc/rc.conf:

moused_enable="YES"
moused_type="ps/2"
moused_port="/dev/psm0"

And start daemon right again:

/etc/rc.d/moused start

X-Window on FreeBSD with NVIDIA GT218 card

Tested: FreeBSD 13.2 on ZFS and another one on UFS (on SSD)

Card:

pciconf -lv | fgrep -B4 VGA

vgapci0@pci0:6:0:0:	class=0x030000 rev=0xa2 hdr=0x00 vendor=0x10de device=0x0a65 subvendor=0x19da subdevice=0x7222
    vendor     = 'NVIDIA Corporation'
    device     = 'GT218 [GeForce 210]'
    class      = display
    subclass   = VGA

Using guide from:

Setup as root

pkg install xorg
pw groupmod video -m USERNAME
pkg install xf86-video-nv

Now relogin as USERNAME and issue startx - this should be enough to start X-Window.

But in my case after startx command there was error in /var/log/Xorg.0.log

(EE) NV: The PCI device 0x10de0a65 (GeForce 210) at 06@00:00:0 has a kernel module claiming it.
(EE) NV: This driver cannot operate until it has been unloaded.
(EE) open /dev/dri/card0: No such file or directory
(WW) Falling back to old probe method for modesetting
(EE) open /dev/dri/card0: No such file or directory

And it used VESA mode... Trying to fix lazy way (kernel moduels are often very sensitive to kernel version mismatch):

pkg install drm-kmod

Ehm, for nvidia there are 2 choices:

Warning! In my case (GT218) I have to use 340 version of driver:

pkg install nvidia-driver-340
# confirm replace of 515 version with older 340 version

After reboot:

sysrc kld_list+=nvidia
kldload nvidia
dmesg

nvidia0: <GeForce 210> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io

pkg install nvidia-xconfig
# run it
nvidia-xconfig

Hmmm:

This server has a video driver ABI version of 25.2 that this
driver does not officially support.  Please check
http://www.nvidia.com/ for driver updates or downgrade to an X
server with a supported driver ABI.

Hmm, so my nVidia card GT218 is supported by old driver only, but it does not support current X-Server...

So my only option was to follow advice (two dashes are used to pass arguments to X-Server):

startx -- -ignoreABI

And yes! it started! :-)

To make ignoreABI permanent we can follow:

Section "Device"
        Identifier "Card0"
        Driver     "nvidia"
EndSection
Section "ServerFlags"
        Option "IgnoreABI" "true"
EndSection

To verify that GLX (OpenGL Extension for X11) is working do this:

pkg install glx-utils

Then run under X11 (as regular user):

glxinfo -B

name of display: :0.0
display: :0  screen: 0
direct rendering: Yes
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 1024 MB
    Total available memory: 1024 MB
    Currently available dedicated video memory: 964 MB
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce 210/PCIe/SSE2
OpenGL core profile version string: 3.3.0 NVIDIA 340.108
OpenGL core profile shading language version string: 3.30 NVIDIA via Cg compiler
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile

OpenGL version string: 3.3.0 NVIDIA 340.108
OpenGL shading language version string: 3.30 NVIDIA via Cg compiler
OpenGL context flags: (none)
OpenGL profile mask: (none)

OpenGL ES profile version string: OpenGL ES 2.0 NVIDIA 340.108 340.108
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 1.00

If there are any errors instead you have problem and proprietary NVidia driver is likely not working.

Installing XFCE environment

Additionally I like XFCE (or LXDE as fallback) so let's try:

# as root
pkg install xfce

See https://docs.freebsd.org/en/books/handbook/desktop/ for full list of available environments.

There are 2 ways how to start X-Window +XFCE4 from local console

  1. using just startxfce4
  2. or use standard startx but first you have to patch global configuration file as shown below:
diff -u /usr/local/etc/X11/xinit/xinitrc{.orig,}
--- /usr/local/etc/X11/xinit/xinitrc.orig	2023-05-01 15:08:45.476207000 +0200
+++ /usr/local/etc/X11/xinit/xinitrc	2023-05-01 15:09:24.455381000 +0200
@@ -48,9 +48,12 @@
 	unset f
 fi
 
-twm &
-xclock -geometry 50x50-1+1 &
-xterm -geometry 80x50+494+51 &
-xterm -geometry 80x20+494-0 &
-exec xterm -geometry 80x66+0+0 -name login
+exec /usr/local/bin/xfce4-session
+exit 1
+
+#twm &
+#xclock -geometry 50x50-1+1 &
+#xterm -geometry 80x50+494+51 &
+#xterm -geometry 80x20+494-0 &
+#exec xterm -geometry 80x66+0+0 -name login

I selected 1st option and did this dirty thing to start XFCE also with startx:

cd /usr/local/bin/
mv startx startx.orig
ln -s startxfce4 startx

Once you are familiar with X-Window environment here is list of may favourite X-apps:

# use firefox-esr if you want rather stable conservative version
pkg install firefox evince-lite

Winamp like mp3 player - will also install Pulse Audio and all Qt5 dependencies (around 100MB):

pkg install qmmp-qt5

NOTE: It will also install Video player mplayer (legend from around 2000 year :-)

Virtualization

Virtualization: bhyve

Bhyve is FreeBSD's custom hypervisor. It is described here:

Unfortunately I can't use bhyve:

kldload vmm
# no error reported, but:
dmesg

SVM: Nested Paging feature not available.
module_register_init: MOD_LOAD (vmm, 0xffffffff82c04580, 0) error 6

Details on my CPU:

CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2000.12-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x40fb2  Family=0xf  Model=0x4b  Stepping=2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x2001<SSE3,CX16>
  AMD Features=0xea500800<SYSCALL,NX,MMX+,FFXSR,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x1f<LAHF,CMP,SVM,ExtAPIC,CR8>
  SVM: NAsids=64

Unforunately the commit that introduced that:

Game over. Theoretically we can still use VirtualBox or Xen...

Virtualization: VirtualBox

Finally I had luck with Oracle VirtualBox, but please ensure that you use only Open Source version (so called "Extension Pack" is PAID for commercial use (!))

VirtualBox setup is described here

Theoretically we can use prepared packages - but there is problem:

  • the kernel module is too old and this command will just crash:
kldload vboxdrv
dmesg

KLD vboxdrv.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/vboxdrv.ko - unsupported file type

Safe way described in manual is to build everything from source, however it would take ages to build gcc12, Qt5 of X11 libraies. So I rather used "hybrid" aproach:

  • fist install package (this will also install useless incomptabile virtualbox-ose-kmod):

    pkg install virtualbox-ose

  • first install these packages to speed-up builds as much as possible (NOTE: I expect that you already installed X11 including libraries as shown in above text):

    make build-depends-list
# install packages from above list
pkg install kBuild

  • now build and install kernel modules only:

    cd /usr/ports/emulators/virtualbox-ose-kmod
# confirm defaults when asked, but uncheck INFO, NLS, EXAMPLES,DOCS
make
# reinstall is needed to uninstall PKG and install Port version
make reinstall

  • now try to manually load module and verify that it does not fail

    kldload vboxdrv
kldstat

# ensure that vobxdrv.ko is listed:
Id Refs Address                Size Name
 1   18 0xffffffff80200000  1f3e2d0 kernel
 2    1 0xffffffff82140000     a4a0 cryptodev.ko
 3    1 0xffffffff8214b000   59dfa8 zfs.ko
 4    1 0xffffffff82b20000    27190 ipfw.ko
 5    1 0xffffffff82b48000    5a0d0 vboxdrv.ko

  • now to ensure that module is loaded on boot (but rather from RC than Loader to avoid fatal panic too early):

    sysrc kld_list+=vboxdrv

And we must follow several instructions:

  • append to /etc/rc.conf:

    sysrc vboxnet_enable="YES"

  • add your unprivileged user to group vboxusers:

    doas pw groupmod vboxusers -m $USER

Reboot your OS...

Start X-Window using startx. There should be new application VirtualBox

NOTE: There is no Host-only network defined. To create it you have to go to File -> Host Network Manager menu and click on Create. This should create new Host-only network.

Finally you should download Debian 11 ISO or some similar Linux distribution, try create and run VM - only running VM will verify that VirtualBox works on your CPU properly.

FreeBSD and GDB

You have to install package first:

pkg install gdb

Now very important thing:

  • you want to put breakpoint to write(2) system call.
  • however libc actually uses __sys_write symbol (the write is weak alias to this internal function)
  • so to properly break at write(2) you have to do this
gdb /bin/ls

...
(gdb) break __sys_write

Function "__sys_write" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (__sys_write) pending.

(gdb) run

reakpoint 1.2, _write () at _write.S:4
4	_write.S: No such file or directory.
(gdb) bt
#0  _write () at _write.S:4
#1  0x0000000801213047 in _swrite (fp=0x8012d3848, buf=0x80183e000 "fbsd13.2-zfs-mbr-fail\ttest-2nd-drive\n", n=37)
    at /usr/src/lib/libc/stdio/stdio.c:119
#2  0x000000080120b15e in __sflush (fp=0x8012d3848) at /usr/src/lib/libc/stdio/fflush.c:130
#3  0x000000080121fe3e in __swbuf (c=10, fp=0x8012d3848) at /usr/src/lib/libc/stdio/wbuf.c:93
#4  0x0000000001028139 in __sputc (_c=10, _p=0x80183e000) at /usr/obj/usr/src/amd64.amd64/tmp/usr/include/stdio.h:468
#5  printcol (dp=0x7fffffffde98) at /usr/src/bin/ls/print.c:386
#6  0x000000000102697f in display (p=p@entry=0x8018360a0, list=0x801836140, options=options@entry=25)
    at /usr/src/bin/ls/ls.c:987
#7  0x0000000001025de5 in traverse (argc=1, argv=<optimized out>, options=37) at /usr/src/bin/ls/ls.c:687
#8  0x0000000001025bdc in main (argc=0, argv=0x0) at /usr/src/bin/ls/ls.c:614
(gdb) quit

Here are references for write vs. __sys_write - not sure which one is most important:

cd /usr/src/lib/libc]
fgrep -r __sys_write

./include/libc_private.h:__ssize_t	__sys_write(int, const void *, __size_t);
./include/libc_private.h:__ssize_t	__sys_writev(int, const struct iovec *, int);
./sys/interposing_table.c:	SLOT(write, __sys_write),
./sys/interposing_table.c:	SLOT(writev, __sys_writev),
./sys/Symbol.map:	__sys_write;
./sys/Symbol.map:	__sys_writev;
./sys/write.c:__weak_reference(__sys_write, __write);
./sys/writev.c:__weak_reference(__sys_writev, __writev);

I used that knowledge to find, why zpool create command wipes disklabel:

Also another issues with bsdinstall:

Building kernel without modules

In my case FreeBSD 13.2 does not need any module and I want to copy build kernel to another machine. To do this I do:

# on build machine
cp /usr/share/examples/etc/make.conf /etc/

Now I edit /etc/make.conf this way:

# on build machine
# uncomment this line and empty list of modules
MODULES_OVERRIDE=

Now we will build kernel traditional way (similar to NetBSD):

# on build machine
cd /usr/src/sys/amd64/conf/
cp GENERIC HP_SATA
vim HP_SATA # now replace "ident GENERIC" with "ident HP_SATA"
config HP_SATA
cd ../compile/HP_SATA/
make cleandepend && make depend
# now just build (should be enough even after code change)
make
scp kernel root@TARGET_MACHINE:/root

Finally backup kernel on Target machine and install new kernel:

# on TARGET machine
cd /boot/kernel
cp kernel kernel.old
cp /root/kernel .

Booting FreeBSD from Linux/GRUB

For long time I was able to boot FreeBSD 13.2 from GRUB only in chainloader mode which is tricky when using GPT+BIOS partitioning. When I tried obvious way - booting FreeBSD kernel using kfreebsd GRUB loader it just reset computer without any helpful message.

However finally I found the trick of the trick on: https://forums.freebsd.org/threads/grub-boot-s.76496/ The key idea is to boot /boot/loader (!) - which is LUA loader for FreeBSD instead of /boot/kernel/kernel (!). Here is example of my Proxmox VE's /etc/grub.d/40_custom entry for FreeBSD:

# FreeBSD is on 1st drive (SSD), GPT+BIOS mode
menuentry "FreeBSD - SSD" {
        # root points to partition holding /boot/grub/i386-pc
        set root='hd0,gpt3'
        insmod ufs2
        insmod bsd
        # now root points to FreeBSD FFS partition
        set root='(hd0,gpt9)'
        kfreebsd ($root)/boot/loader
        boot
}

FreeBSD as guest (VM)

  • 2023 - confirmed that FreeBSD 14.0 (plain ISO) works fine Generation 2 (UEFI, Secure boot Off) under Hyper-V Role of Windows Server 2012R2 (last version of Hyper-V that works on my AMD Opteron)

Conflict of MariaDB vs. perl-DBD packages

There is serious bug in PKG system, it is not possible to install both MariaDB and suitable perl-DBD package. These commands will fail with conflicts:

# does not work
pkg install mariadb106-server p5-DBD-mysql
# also does not work
pkg install mariadb106-server p5-DBD-MariaDB

See

2nd point advises to add to /etc/make.conf:

DEFAULT_VERSIONS+= mysql=105m

and build perl DBD package from ports (did not test it yet)

Chain loading from FreeBSD boot loader

FreeBSD bootloader now supports chain-loading of another system:

Example scenario:

  • I have MBR/BIOS partition setup on old (but still working) Maxtor 200GB SATA 1 disk connected to add-on AHCI PXIe card (ASMedia ProductID=0x1164).
  • 1st primary partition is FreeBSD 14.1
  • 2nd primary partition is NetBSD 10
  • 3rd primary partition is OpenBSD 7.5

Here is MBR layout as seen by fdisk sd0 run under OpenBSD 7.5 (3rd partition 2: with id=0xA6):

Disk: sd0	geometry: 24792/255/63 [398297088 Sectors]
Offset: 0	Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
*0: A5      0   1   2 -   8485  54  17 [          64:   136314880 ] FreeBSD
 1: A9   8485  85  49 -  16771 184  33 [   136316928:   133120812 ] NetBSD
 2: A6  16771 184  34 -  24792 215  63 [   269437740:   128859348 ] OpenBSD
 3: 00      0   0   0 -      0   0   0 [           0:           0 ] Unused

To boot NetBSD 10 instead of FreeBSD I do this:

  • press ESC to get prompt from FreeBSD loader
  • invoke lsdev to see disk devices and partitions
  • then specify chain DISK_NAME: - notice that you must use full name including trailing colon (:)
  • my example (again note colon!): 
    chain disk0s2:
  • similarly to boot OpenBSD one has to enter chain disk0s3: command in FreeBSD bootloader.

DTrace

DTrace is dynamic tracing framework similar to Linux perf. It allows to hook various places in both kernel and userspace and find possible bottlenecks. Detailed guide is in official handbook at https://docs.freebsd.org/en/books/handbook/dtrace/

Here is really quick quick start:

kldload dtraceall
pkg install dtrace-toolkit

DTrace example scripts are under:

  • /usr/share/dtrace/ (included with base system)
  • /usr/local/share/dtrace-toolkit/ (installed as package)

Now you can try, for example:

/usr/local/share/dtrace-toolkit/hotkernel

Recommended approach is:

  • first try top -a, vmstat 1, iostat 1 or systat -vmstat (exit with :q) to know where is unexpected system load (is it user% CPU, system% CPU or load average?)
  • typically you will find that something is causing high system% CPU usage
  • unless iostat 1 shows clue that it is disk I/O you basically exhausted usable information from system tools
  • now you should try DTrace scripts like hotkernel to find more...

WARNING! Unfortunately DTrace has similar issue as Linux BPF/BCC - kernel changes so often that some scripts will simply no longer work:

/usr/local/share/dtrace-toolkit/Disk/iofile.d

dtrace: failed to compile script \
  /usr/local/share/dtrace-toolkit/Disk/iofile.d: \
  line 54: probe description io:::wait-start does not match any probes

Resources

Copyright © Henryk Paluch. All rights reserved.

Creative Commons License This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License

⚠️ **GitHub.com Fallback** ⚠️