Trying to streamline znapzend deployment.

There are a couple of key points to keep in mind:

• source datasets, upon zpool creation, should be case-insensitive
• the target datasets on the MacOS daily-driver machine have the zfs readonly flag set (see below).

• there must be an "environment" file under ~/.ssh in a target MacOS machine for it to accept znapzend connections
• this way, an ssh environment (that includes $PATH) is provided when connecting
• here's what it may contain

PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/usr/local/zfs/bin

Reference: https://github.com/oetiker/znapzend/issues/327 and https://superuser.com/questions/1484960/set-path-when-ssh-into-macos

• furthermore, there should be a mechanism for caffeinating ssh connections, as follows

<username>@<target-mac> ~ % cat /etc/ssh/sshd_config.d/200-<target-mac>.conf

ClientAliveInterval 0
ClientAliveCountMax 8
MaxStartups 10:30:100
# must disable this before copy-id from host to here
ForceCommand /usr/local/bin/sshd_caffeinate 

Note that the "sshd_caffeinate" script is provided in the Appendix section.

The ~/.ssh/config should contain the following:

Match host <target-mac>.local exec "/usr/local/bin/wakeonlan f0:a0:10:80:10:60 &"

Host <target-mac>.local
Hostname <target-mac>.local
User <username>
PubKeyAuthentication yes
StrictHostKeyChecking no
IdentityFile /Users/<username>/.ssh/id_ed25519
SendEnv LANG LC_*
PermitLocalCommand yes
LocalCommand caffeinate -w $PPID &

The (homebrew) daemon arguments in /Library/LauchDaemons are as follows (i.e. default):

	<key>ProgramArguments</key>
	<array>
		<string>/usr/local/opt/znapzend/bin/znapzend</string>
		<string>--connectTimeout=120</string>
		<string>--logto=/usr/local/var/log/znapzend/znapzend.log</string>
	</array>

• added some /etc/sshd_config settings and a ForceCommand script so that the target MacOS machine does not sleep during zfs and rsync transfers (it seems that they do not create a wake assertion as one would expect): refer to the zAppendix section

Everything is set up properly, and works as intended.

• the source and target MacOS computers are in different apartments, with two individual FritzBox! routers: one main, and one attached as a wireless bridge (in a so-called "Mesh"), with an ethernet connection to the target (to provide wake-on-lan)
• ssh setup is complete, for passwordless login and with WOL capability
• rsync also works, had some difficulty with stalls over wifi
• znapzend obviously works

Updated the crontab on a Proxmox server for rsync backups as follows:

/usr/sbin/etherwake -i enp6s0 ax:ax:5x:bx:7x:bx && /usr/sbin/etherwake  -i enp6s0 ax:ax:5x:bx:7x:bx && sleep 1 && /usr/bin/rsync -a --no-perms --no-owner --no-group --log-file=/var/log/rsync-devo/"$(date +\%F_\%I_\%M)"_datasetname.txt --delete  --exclude='.DocumentRevisions-V*' --exclude='.AppleDouble' --exclude='.fseventsd' --exclude='.metadata_never_index' --exclude='.Spotlight-V*' --exclude='.Trashes' --exclude='.TemporaryItems'  /zpoolname/datasetname/ username@macos_destination.local:/Volumes/zpooltarget/datasetname

The changes include: some logging

Regarding the zfs datasets that are updated using rsync (from a Proxmox server), and not znapzend:

• set atime=on
• updated the crontab rsync command as follows:

0 5 * * Mon /usr/sbin/etherwake -i enp6s0 ax:ax:5x:bx:7x:bx && /usr/sbin/etherwake  -i enp6s0 ax:ax:5x:bx:7x:by && sleep 10 && /usr/bin/rsync -a --log-file=/var/log/rsync-logs/"$(date +\%F_\%I_\%M)"log.txt --delete  --exclude='.DocumentRevisions-V*' --exclude='.fseventsd' --exclude='.metadata_never_index' --exclude='.Spotlight-V*' --exclude='.Trashes'  /zpoolname/datasetname/ [email protected]:/Volumes/zpoolosx/zfsdataset

• good news: everything is working!
• the main MacOS machine sends four backup jobs via znapzend, to the (zfs set readonly) target pool
• a simple Applescript creates four read/write zfs clones, with full editing ability; running it again will update the clone from the latest znapzend backup
• znapzend keeps on hugging along
• this has been achieved by rethinking the approach, as outlined in the previous note

• updated to Sonoma 14.5 (host and destination) and re-enabled Spotlight
• of note, the PermitUserEnvironment flag in the destination /etc/ssh/sshd_config was reset to default (!), so must be re-enabled; otherwise, znapzend will throw "zfs command could not be found" errors
• total rethink of the approach, as there had been a serious, obvious (in retrospect) design error: the target datasets in the MacOS destination were active, i.e. allowed interaction, unlike the Linux server datasets which were for backup purposes only
• for this reason, znapzend's destination on an active MacOS workstation should have the zfs readonly flag set; this allows proper zfs send/receives and snapshot management; other solutions (e.g. setting atime=off) simply don't work
• as for actually using the target datasets on the MacOS workstation, one has to clone them, which is quite easy to do and permits full read-write access
• a simple AppleScript is provided in the repo (but must be modified, as the dataset names are hardcoded)

More changes:

• the daemon is configured as follows:

    <string>/opt/znapzend-0.22.1/bin/znapzend</string>
    <string>--connectTimeout=120</string>
    <string>--features=oracleMode,skipIntermediates</string>
  

• added the following to the /etc/ssh/ssh_config file, to issue a caffeinate command when a ssh connection is started (credit goes to this amazing blog post

PermitLocalCommand yes

LocalCommand caffeinate -w $PPID &

• simplified the backup jobs (removed "strategically placed" caffeinate commands
• saw zfs recv -F hangs after the above settings, so added (as per this post, unclear whether related or not)

sudo zfs set primarycache=metadata

sudo zfs set secondarycache=metadata

• changed the backup jobs to "recursive:off"

• changed the daemon (in /Library/LaunchDaemons), to include the following features:

   <array>
            <string>/opt/znapzend-0.22.1/bin/znapzend</string>
            <string>--connectTimeout=120</string>
            <string>--features=compressed,oracleMode</string>
  </array>
  

• edited all backup jobs to include strategically placed caffeinate and killall caffeinate commands (the source machine should sleep properly, and so should the target)

• installed the 2.2.3 release candidate version of OpenZFSonOSX, on both source and target
• removed Homebrew's version of znapzend (and perl)
• installed znapzend from the Github page (and used the vanilla daemon, without special --features)
• disabled Spotlight on the source datasets (had an .mds process-related kernel panic on the target, which mirrors the source settings)
• the procedure to disable Spotlight is outlined here, by Lundman

• removed OpenCore Legacy Patcher, because, in my experience, it introduces instability
• upgraded to OpenZFSonOSX 2.2.3 release candidate 4 (the Catalina .pkg)
• stopped the daemon temporarily, with sudo brew services stop znapzend
• edited the znapzend daemon .plist (see the procedure for the HomeBrew package here), to include these optional features: compressed and SkipIntermediates

<key>ProgramArguments</key>

<array>
	<string>/usr/local/opt/znapzend/bin/znapzend</string>
	<string>--connectTimeout=120</string>
	<string>--features=compressed,skipIntermediates</string>
	<string>--logto=/usr/local/var/log/znapzend/znapzend.log</string>

• did a "service" run to update all datasets, using the --runonce command:

znapzend --features=compressed,skipIntermediates --runonce=<znapzend source>

• restarted the daemon with sudo brew services start znapzend

• added these options in the sender's /etc/ssh/ssh_config:

ServerAliveInterval 240

ServerAliveCountMax 10

• commented out the following

# SendEnv LANG LC_*

• RestrictEvents.kext, with the revblock=media boot argument to disable mediaanalysisd process (suspected source of instability/hangs when using legacy Metal 1 GPUs, running under OCLP)
• the current boot arg is: alcid=12 debug=0x100 keepsyms=1 swd_panic=1 -ctrsmt -wegnoigpu revpatch=sbvmm revblock=pci,media -mausiwol
• unplugged the port running IntelMausi.kext, and used the 2.5Gb port in the AsRock Z690M-ITXax motherboard, along with the LucyRTL8125Ethernet driver
• June 23, 2024 update: could not get the 2.5Gb Ethernet port to wake-on-lan, so we're back using IntelMausi on the Intel port