When I got the boards I soldered the module as well as the two connectors and hooked the JTAG up to my FT2232H breakout board as the official XDS110 probes from TI are currently unavailable due to a chip shortage. Theoretically this should work as OpenOCD has support for both the probe as well as the SoC. But with neither OpenOCD 0.11 nor 0.12-rc3 the JTAG chain even probed correctly. Here I should note that TI has made the (questionable) design decision to only support the relatively-uncommon cJTAG (a 2-wire variant of JTAG) by default, requiring a set of specific scan codes sent into the cJTAG interface to actually switch into normal 4-wire JTAG operation. This evidently didn't work as I didn't see any activity on the TDI wire. After lots of googling I eventually stumbled upon issue #375 in the OpenOCD issue tracker which recommended adding runtest 20 before the cJTAG to JTAG transition sequence. This just runs 20 idle cycles on the JTAG interface before starting the transition, essentially JTAG's equivalent of putting a sleep into code. Patching this into my OpenOCD configuration made the JTAG chain work immediately.

But I wasn't done yet. While the SoC now identified correctly on JTAG, the ARM debug adapter (DAP), needed to access the flash, failed to initialize. I reliably got Error: Invalid ACK (4) in DAP response which is a rather unhelpful error, even googling results in no useful results. At this point I tried to use this setup with a different CC2652P2 on another board I own, which worked perfectly. So at this point it's not an issue with my JTAG adapter or OpenOCD configuration, but the SoC itself seems to either be blocking access to the ARM debug adapter or the core together with the debug adapter is powered down. This was a bit of a suprise considering that this is a brand-new module and thus should theoretically be clean and have no firmware on it. My best guess is that this firmware is left over from factory testing. Now, most SoCs have a feature where one can essentially factory-reset the entire chip, disabling any debug protection but also deleting all firmware on the chip.

These TI chips do in fact have a mass erase feature, though one needs to look in the 1700-page reference manual to find it. Under section 5.8 Debug Features Supported Through WUC TAP there is the CHIP_ERASE_REQ operation which does erase everything from the chip. Before I can use this functionality however I first need to be able to access the Wake Up Controller Test Access Point, which is a dedicated JTAG TAP. TI has their own JTAG router on these chips, which they call ICEPick. In Table 5-5 of the reference manual there is an overview of the various TAPs available to be enabled in the JTAG router. The TAP in question is called "AON WUC" here, available on the 5th test bank. OpenOCD recently (in unreleased verison 0.12) got support for not just enabling the debug banks but also the test banks through ICEPick. Enabling TAPs is done with icepick_c_tapenable followed by the TAP for the ICEPick itself, followed by the bank index. Debug banks start from 0, test banks from 16. So test bank 5 is at index 21. I guessed the instruction register length (irlen) based on the documented commands. The complete WUC TAP definition looks like this:

jtag newtap $_CHIPNAME wuc -irlen 4 -ircapture 0x1 -irmask 0xf -disable
jtag configure $_CHIPNAME.wuc -event tap-enable "icepick_c_tapenable $_CHIPNAME.jrc 21"

Now I have a JTAG TAP for the Wake Up Controller and can finally issue the documented commands. The IR value (0x01) needs to be shifted into the instruction register using irscan, the data register needs to be filled with the documented command bit and shifted in using drscan. After each command I run another 20 idle cycles just to make sure that the previous command is executed. This might not be necessary, but it takes negligible time. So issuing the mass erase looks like this:

# CHIP_ERASE_REQ
irscan $_CHIPNAME.wuc 0x01 -endstate IRPAUSE
drscan $_CHIPNAME.wuc 8 0x2 -endstate DRPAUSE
runtest 20

# MCU_VD_RESET_REQ
irscan $_CHIPNAME.wuc 0x01 -endstate IRPAUSE
drscan $_CHIPNAME.wuc 8 0x20 -endstate DRPAUSE
runtest 20

After running this, I tried connecting with my regular debug configuration again and the DAP came up immediately. So the chips were indeed not clean and that was the reason the DAP didn't connect.

If you want a complete solution for the CC2652, here is a OpenOCD script combining everything:

source [find target/icepick.cfg]
source [find target/ti-cjtag.cfg]

if { [info exists CHIPNAME] } {
        set _CHIPNAME $CHIPNAME
} else {
        set _CHIPNAME cc26x2
}

#
# WUC TAP
#
jtag newtap $_CHIPNAME wuc -irlen 4 -ircapture 0x1 -irmask 0xf -disable
jtag configure $_CHIPNAME.wuc -event tap-enable "icepick_c_tapenable $_CHIPNAME.jrc 21"

#
# ICEpick-C (JTAG route controller)
#
if { [info exists JRC_TAPID] } {
        set _JRC_TAPID $JRC_TAPID
} else {
        set _JRC_TAPID 0x3bb4102f
}
jtag newtap $_CHIPNAME jrc -irlen 6 -ircapture 0x1 -irmask 0x3f -expected-id $_JRC_TAPID -ignore-version
jtag configure $_CHIPNAME.jrc -event setup "jtag tapenable $_CHIPNAME.wuc"
# A start sequence is needed to change from 2-pin cJTAG to 4-pin JTAG
jtag configure $_CHIPNAME.jrc -event post-reset "ti_cjtag_to_4pin_jtag $_CHIPNAME.jrc"

init

# CHIP_ERASE_REQ
irscan $_CHIPNAME.wuc 0x01 -endstate IRPAUSE
set _res [drscan $_CHIPNAME.wuc 8 0x2 -endstate DRPAUSE]
echo [format %x $_res]
runtest 20

# MCU_VD_RESET_REQ
irscan $_CHIPNAME.wuc 0x01 -endstate IRPAUSE
set _res [drscan $_CHIPNAME.wuc 8 0x20 -endstate DRPAUSE]
echo [format %x $_res]
runtest 20

reset_config srst_once
adapter srst delay 100

exit

This can then be used from either the command line or another script loading this one. This is mine:

source [find interface/ftdi/minimodule.cfg]
transport select jtag
adapter speed 100
source ./cc26x2-mass-erase.cfg

I went ahead and replaced the module in my notebook and started it. Upon booting, the firmware printed "Unauthorized Wireless network card is plugged in. Power off and remove it" and refused to do anything after that. This is because Lenovo has a module built into the firmware which checks if the WiFi module has the exact same PCI ID as the one originally shipped and aborts the boot if it detects any other ID.

A workaround (at least on Linux) is to boot with the original module installed (otherwise the PCIe port doesn't get initialized by the firmware), delete the PCI device for the module (echo 1 > /sys/class/pci_bus/[your_bus_num]/device/delete), suspend the notebook, replace the module while in suspend, resume and rescan the bus (echo 1 > /sys/class/pci_bus/0000:00/rescan). You have to do this every time you reboot though, which makes this rather tedious. On Linux you can use kexec to soft-reboot without going through the firmware, but there are still things which need a hard reboot.

To actually fix the problem at the root, I need to patch the firmware which is stored on a flash chip on the motherboard. There are two ways to get to it, via the chipset SPI controller while the machine is booted or from an external clip probe while the machine is without power. The first method does not require opening the notebook nor additional hardware but is risky. If the modified firmware you flash has any issues booting you essentially have a brick unless you have the external clip probe and flasher for the second method.

I have the necessary hardware, so I decided to go with the safer method. I disassembled the notebook until I got access to the SPI flash chip containing the firmware. This is in 99% of cases a single SOIC-8 chip with 25 somewhere in the part number. For the Lenovo Yoga 2 Pro this is a W25Q64FVSSIG, a 3V 64Mbit flash chip made by Winbond. I used a CH341a-based flasher set to 3.3V (make sure that you're not feeding too much voltage into your IC) with a SOIC-8 clip probe to access it. On the software side I used flashrom, which has support for tons of SPI flash chips including this one. I first read the existing contents with flashrom -p ch341a_spi --read=stock.rom, then did the same thing again and compared checksums to make sure I have a valid copy of the stock contents should anything go wrong. With the flash contents in hand, I needed to hack out the check.

Since this notebook, same as most since ~2011 has UEFI-based firmware, I used UEFITool to open the flash contents. It successfully detects a bunch of data structures, including two UEFI FFSv2 filesystems. I extracted the complete body of both filesystems and called strings on it to find the one containing the messge I was seeing. Since UEFI contains both single-byte-based ASCII/UTF-8 text as well as UCS2/UTF-16 text I needed to call it with strings -e l as well (note UEFI is always little-endian). I spotted the error message string in the second strings call of the first FFSv2 volume in close proximity to the OneKeyRecovery executable as well as the UEFIL05BIOSLock executable. I decided to extract the PE body of both (in UEFITool) and called strings on both again, determining that UEFIL05BIOSLock is the executable containing the WiFi module authorization code. Since this executable is just a special Windows PE file, I loaded it into Ghidra to figure out how it operates.

I searched Ghidra for the error message (Search -> For strings) and jumped to the sole function it is referenced from, which I called checkHandler:

/* WARNING: Globals starting with '_' overlap smaller symbols at the same address */

void checkHandler(EFI_EVENT *Event,void *context)

{
  EFI_STATUS status;
  bool consoleCleared;
  bool stopBoot;
  
  stopBoot = false;
  consoleCleared = false;
  if ((enableL05WWANLock) && (_wwanState == 2)) {
    clearConsole();
    consoleCleared = true;
    print(L"\nUnauthorized WWAN network card is plugged in. Power off and remove it");
    stopBoot = true;
  }
  if ((((enableL05WLANLock != false) && (wlan_pci_buses != (wlan_pci_bus_desc *)0x0)) &&
      (authorized_wlan_devices != (authorized_wlan_device *)0x0)) &&
     (status = checkWLANAuthorized(), status == 0x8000000000000003)) {
    if (!consoleCleared) {
      clearConsole();
    }
    print(L"\nUnauthorized Wireless network card is plugged in. Power off and remove it");
    stopBoot = true;
  }
  if (stopBoot) {
    do {
    } while( true );
  }
  return;
}

Note that this is the fully reverse-engineered function, after I've given everything names and fixed any issues with data structures. This is very different from what you get when you're just opening this executable in Ghidra. If you find yourself needing to reverse-engineer (U)EFI executables, I can highly recommend uefi-firmware-tools by Alex James which I also used to assist to reverse-engineer this module.

In a nutshell what this function does is it checks if the WWAN (mobile network) or WLAN lock global variables are set, if they are it checks if the authorized device global variables aren't null pointers and if that succeeds it tries to check if the device is authorized. In case of the WWAN device the check has already been performed prior to this function being called and its result is stored in _wwanState. If either the WWAN or WLAN device are not in the authorized list it sets stopBoot to true and prints the aformentioned message to the screen. Finally it just enters an infinite loop to prevent the system from booting.

The actual check is in checkWLANAuthorized which is a relatively lengthy function which I'm not going to show here reading the PCI device config space for each bus in a list of buses, reading the vendor and product IDs from each of those devices and comparing it against a list of authorized devices.

checkHandler is referenced from the executable entry point (the low-level equivalent of a main function), where it gets passed to a function (efiRegisterNotify) which registers it as an EFI event handler to be executed just before the boot. In this function one can also see where most of the data in global variables is coming from: A custom Lenovo EFI protocol with GUID a98e0db6-796e-4c19-b3a23036ad5f02a7 containing a pointer to a rather large data structure with the list of approved devices as well as the bus identifiers (PCIe for WiFi, USB for WWAN) which need to be checked.

EFI_STATUS _ModuleEntryPoint(EFI_HANDLE ImageHandle,EFI_SYSTEM_TABLE *SystemTable)

{
  EFI_STATUS status;
  undefined local_res18 [16];
  
  setHandles(ImageHandle,SystemTable);
  enableL05WLANLock = enableL05WLANLock;
  enableL05WWANLock = enableL05WWANLock;
  if ((lenovo_ext_con_proto == 0) &&
     (status = (*bootServices->LocateProtocol)
                         (&EFI_GUID_180001a50,(void *)0x0,(void **)&lenovo_ext_con_proto),
     (longlong)status < 0)) {
    return status;
  }
  // Lenovo's authorized devices protocol
  // GUID a98e0db6-796e-4c19-b3a23036ad5f02a7
  status = (*bootServices->LocateProtocol)(&lenovo_auth_proto_guid,(void *)0x0,&lenovo_auth_proto);
  if ((longlong)status < 0) {
    status = 0;
  }
  else {
    if (enableL05WLANLock != false) {
      wlan_pci_buses = lenovo_auth_proto->auth_data->wlan_pci_buses;
      authorized_wlan_devices = lenovo_auth_proto->auth_data->authorized_wlan_devices;
    }
    if (enableL05WWANLock != false) {
      wwan_usb_buses = lenovo_auth_proto->auth_data->wwan_usb_buses;
      authorized_wwan_devices = lenovo_auth_proto->auth_data->authorized_wwan_devices;
      if ((wwan_usb_buses != (undefined **)0x0) &&
         (authorized_wwan_devices != (authorized_wwan_device *)0x0)) {
        registerWWANCheck();
      }
    }
    status = efiRegisterNotify(8,checkHandler,0,local_res18);
  }
  return status;
}

void efiRegisterNotify(EFI_TPL template,EFI_EVENT_NOTIFY notify_func,void *ctx,EFI_EVENT *event)

{
  if (notify_func == (EFI_EVENT_NOTIFY)0x0) {
    notify_func = (EFI_EVENT_NOTIFY)&DAT_180000934;
  }
  (*bootServices->CreateEventEx)(0x200,template,notify_func,ctx,&EfiEventReadyToBootGuid,event);
  return;
}

So now that we understand how this executable works, I needed to find a way to bypass it. The logically simplest way would be to just delete it, but it creates custom events in functions not shown here, the absence of these events might cause unknown problems and I don't have the time to reverse-engineer the rest of the firmware to make sure it doesn't. Another way would be to add my new WiFi module's IDs to the list of approved devices, but that list is not in here and this would require further reverse-engineering. Or I could just use my knowledge of this executable to hack it so it never triggers the infinite loop.

In this case I'm actually quite lucky as the global variables enableL05WLANLockStatic and enableL05WWANLockStatic are statically defined in this executable's data section which I can modify. Setting both of those to zero would essentially bypass all of this module's harmful features as none of the checkHandler's conditions would evaluate to true.

They are also in a neat OEM Switch section, meaning they are likely intended to be toggled by the OEM while testing and toggling them thus should not cause any adverse effects. Because they have a nice string description just in front of the values (be aware of the trailing null bytes also in front of them though) I chose to just open the executable in a hex editor, find the two relevant strings and changed the two bytes from 0x01 to 0x00, thus disabling the lock. To confirm the mod, one could load the resulting binary back into Ghidra but I was quite confident in my change, so I just went ahead and used UEFITool again to replace the PE body of the executable in question with the modded one. Note that if the menu entry to do that is greyed out, make sure you use the old engine version of UEFITool to perform the mod as the new engine cannot yet replace parts of the firmware.

After saving the resulting firmware file from UEFITool I used flashrom again to write back my changes to the SPI flash chip, reassembled the notebook and booted it. And indeed, it booted straight into the operating system, even with the "unauthorized" WiFi module installed!

Now, could I have guessed how to patch this just from the strings output and a bit of trial & error? Most likely, but to me it's a lot more fun and educational to solve the problem a bit more thoroughly.

A few notes for last:

1. I am in no way responsible for anything you do to your devices as a result of this blog post. This is explicitly unsupported by the vendor and you do it on your own risk.
2. This post has been written specifically using the Lenovo Yoga 2 Pro's firmware. Please exercise caution and don't blindly apply it to other models.

The way Linux knows what devices are present and should be exposed on a given SBC is through a Device Tree, a piece of data which describes all hardware exposed on a given SBC. For a lot of the more popular SBCs (like the ROCK64) these are maintained by the Linux kernel developers in the main source tree. By convention there are usually at least two device tree source files which describe a board: One describing the peripherals the SoC (System on Chip) has and one describing which ones are wired up on a particular board and to what. Linux's device trees have a policy of not enabling peripherals if nothing is connected to them on a board, so if you wire up hardware to your SBC, you most likely need to extend the device tree to cover that extra hardware.

Looking at the ROCK64 Device Tree we can see that on it the spi0 controller being enabled and configured:

 &spi0 {
    // Enable the spi0 controller (remember the SoC device tree disables all peripherals just connected to the outside by convention)
	status = "okay";

    // Define a flash device on Chip Select 0 (the number after the @)
	flash@0 {
        // What driver to load for this device
		compatible = "jedec,spi-nor";
        // reg is the Chip Select again and must match the number after the @ of the definition.
		reg = <0>;
        
        // ...
	};
};

So this tells us that the ROCK64 board uses the spi0 controller from the SoC and has a SPI NOR flash fitted which occupies Chip Select 0. So to connect my device I either need to use a different SPI controller (spi1) or a different Chip Select on spi0. As it turns out spi1 is not available on the ROCK64 because the RK3328 doesn't have enough pins (there's also the 3368 and 3399 with more), so my only option is to use a second chip select on spi0.

In device tree form this looks like this:

&spi0 {
	status = "okay";
	// Tell the SPI controller that the second CS is used
	num-cs = 2;
    
	// Pins for the SPI controller, documented below
	pinctrl-0 = <&spi0m2_clk &spi0m2_tx &spi0m2_rx &spi0m2_cs0 &spi0_cs1>;
    
	flash@0 {
        // ...
	};
	spidev@1 {
		compatible = "linux,spidev";
		// Enable our spidev
		status = "okay";
		// Again should match the CS line (@1 in this case)
		reg = <1>; 
		// The maximum frequency the SPI slave supports
		// (and your wiring too, floating wires respond poorly to high frequencies).
		spi-max-frequency = <10000000>;
	};
};

I told the SPI controller that I now need two of its SPI chip select lines and defined my spidev on chip select 1. There is one complicated part about this, and that is pinctrl-0. This tells the SPI controller which pins to use and tells the pin controller to switch them to the correct mode. The Linux spi0 definition is actually the same, except that I added spi0_cs1, which is the additional chip select that I'm using for my device. Now here it gets weird. The RK3328 has three GPIO map modes (m0, m1 and m2) for the SPI peripheral. I need to run spi0 in m2 because otherwise the SPI flash doesn't work because it is wired to the pins as mapped in M2 mode. Also for both other modes not all pins are on the physical connector. But there is no spi0m2_cs1 on the chip. So there aren't actually two chip selects available on the board. So I'm stuck.

Or am I? Linux has this nice feature called cs-gpio, which is essentially just using GPIOs as chip selects. Instead of having the hardware SPI controller control the chip select line, Linux does it via GPIO. This is obviously slower, but the chip select line is very infrequently toggled so in practice this doesn't really slow things down. Now according to the documentation for this feature, the resulting device tree should look something like this:

&spi0 {
	pinctrl-0 = <&spi0m2_clk &spi0m2_tx &spi0m2_rx &spi0m2_cs0 &spi0_cs1>;
	// Use the native CS and as a second one GPIO3 pin 7 (GPIO3_A7) as as ACTIVE_LOW (1)
	cs-gpios = <0>, <&gpio3 7 1>;

	spidev@1 {
		compatible = "linux,spidev";
		status = "okay";
		reg = <1>;
		spi-max-frequency = <10000000>;
	};
}
// ...
&pinctrl {
	// ...
	spidev1 {
		spi0_cs1: spi0-cs1 {
        	// GPIO 3 Pin 7 Mode GPIO (0), pull up
 			rockchip,pins = <3 7 0 &pcfg_pull_up>;
 		};
 	};
};

One more thing I need to change is that I've been using the linux,spidev compatible on my spidev. This is however no longer supported on recent Linux versions due to a questionable design decision that each userspace device needs its own compatible string upstreamed into Linux. As this is a hobby project, I'm just going to hijack one already on the list, like cisco,spi-petra. If you're building something that's going to ship to millions you might want to actually get your own compatible string into Linux.

On most SPI controllers I'd now be done. But as it turns out the Rockchip SPI controller is kind of broken. It accepts this and happily gives me /dev/spidev0.1. But there is one big problem: If I'm talking to my spidev, both chip select lines go low. This is very bad and could damage hardware because the bus can be occupied by two devices at the same time. From looking at the code this is a hardware limitation of the Rockchip SPI controller. It needs to always toggle its native chip select, otherwise it doesn't make progress. So I'm stuck again.

Luckily there is an (admittedly kind of hacky) solution to this as well. Let's just use two GPIO chip selects and disconnect the native one. Then the SPI controller can happily toggle the native chip select, it is not connected to anything anymore.

&spi0 {
	cs-gpios = <&gpio3 8 1>, <&gpio3 7 1>;
};
&pinctrl {
	// ...
    spi0-2 {
		// ...
		// Override definition of spi0m2_cs0
		spi0m2_cs0: spi0m2-cs0 {
			// GPIO3 Pin 8/B0 Mode GPIO (0), pull up
			// Note the mode change from 4 to 0, this switches the pin from being controlled by the SPI controller to a GPIO
			rockchip,pins = <3 8 0 &pcfg_pull_up>;
		};
	};
	spidev1 {
		spi0_cs1: spi0-cs1 {
        	// GPIO 3 Pin 7 Mode GPIO (0), pull up
 			rockchip,pins = <3 7 0 &pcfg_pull_up>;
 		};
 	};
};

And with that, I can talk to both devices without issues!

Now for the last part, making this composable. Until now I've used excerpts of the full device tree to document what I did. This is however a relatively poor way of implementing something like this as I'd have to update my device tree for every change the Linux developers make. Let's instead use a device tree overlay where I just overlay my device tree changes on top of the normal device tree.

And this is it:

/dts-v1/;
/plugin/;
/ {
	compatible = "pine64,rock64", "rockchip,rk3328";
	fragment@0 {
		target = <&spi0>;
		__overlay__ {
			#address-cells = <0x1>;
			#size-cells = <0>;

			pinctrl-0 = <&spi0m2_clk &spi0m2_tx &spi0m2_rx &spi0m2_cs0 &spi0_cs1>;
			
			cs-gpios = <&gpio3 8 1>, <&gpio3 7 1>;

			spidev@1 {
				// Hijack random compatible as I have no desire to rebuild Linux to
				// include a custom device ID in spidev's compatible list.
				compatible = "cisco,spi-petra";
				status = "okay";
				reg = <1>;
				spi-max-frequency = <10000000>;
			};
		};
	};
	fragment@1 {
		target = <&pinctrl>;
		__overlay__ {
			spidev1 {
				spi0_cs1: spi0-cs1 {
					rockchip,pins = <3 7 0 &pcfg_pull_up>;
				};
			};
		};
	};
	fragment@2 {
		target = <&spi0m2_cs0>;
		__overlay__ {
			rockchip,pins = <3 8 0 &pcfg_pull_up>;
		};
	};
};

It looks very similar to the last device tree, except that the changes are encapsulated in fragments, a compatible is added to indicate for which boards this overlay is and cell geometry (#address-cells, #size-cells) is added because it cannot currently be inherited from the base device tree. Applying this overlay is highly bootloader- and distro-specific, so I'm leaving that part out of this post.

The commit-graph file is a supplemental data structure that accelerates commit graph walks. If a user downgrades or disables the core.commitGraph config setting, then the existing ODB is sufficient.
...
The commit-graph file stores the commit graph structure along with some extra metadata to speed up graph walks.

So it is basically a persisted cache to speed up certain operations. There is even a command to write a new commit graph,  git commit-graph write. Theoretically this should write a new, valid commit graph. But sadly it fails with the exact same error about the existing commit graph being broken. It appears that we need to get rid of the broken commit graph before generating a new one as the broken one interferes with the generation of the new one.

After some investigation into how Git stores its commit graphs there are two ways they can be stored in a repository (the .git directory unless the repository is bare). There can either be a single commit graph stored at objects/info/commit-graph or a newline-separated list of commit graph hashes at objects/info/commit-graphs/commit-graph-chain. By removing both of those files (most likely only one of them will exist) Git will no longer have a commit graph cache and should work normally again. Optionally you can now write a new commit graph with git commit-graph write, but for example GitLab automatically creates a new commit graph for you when running housekeeping for a repository.

Annex: Investigating the broken commit graph

The affected repository was stored on ZFS so filesystem corruption is very unlikely. When looking at where the error is triggered in Git source code, it appears that Git is hitting a commit date offset with the CORRECTED_COMMIT_DATE_OFFSET_OVERFLOW flag set, which according to its commit message means that the commit date offset exceeds 2³¹ seconds (~68 years). These date offsets are stored in a special block (GDOV) which is not present in the broken commit graph. This means that Git cannot recover the original commit date offset as it would be stored in the GDOV block and that's why it aborts.

Considering that this is a fairly normal repository it seems unlikely that such a date offset would be present and a quick check of all commits confirms that indeed no such offset is present. Also the new commit graph has no GDOV block and works just fine. So what happened? A binary diff of the broken and the new commit graph shows little differences except for a section which is empty (all zero bytes) in the new commit graph and just counts up from 0x800000 bytes (2^31 in binary) to 0x8000009D with a few 0x00000000 sprinkled in between. Sadly the binary format is relatively compact and we cannot get Git to decode it easily because it's broken so I do not know exactly what that section was supposed to be. But it seems clear that Git was interpreting the 31st bit as the offset overflow flag.

I could investigate further and parse the file fully, but I think I'll end it here. It seems likely that this was either caused by a freak accident (bit flip, ...) or a bug in an older version of Git when writing the commit graph.

If anyone else wants to parse the files I've attached both the newly-written and the broken graph files.

Update: Root Cause found

Shortly after this article was written, Will Chandler on the Git mailing list figured out the actual cause of this issue. See https://public-inbox.org/git/DD88D523-0ECA-4474-9AA5-1D4A431E532A@wfchandler.org/ for his write-up. It turned out to be a bug in Git where an upgrade of the commit graph data from v1 to v2 caused an underflow, which flips the overflow flag making the commit graph unreadable.

Luckily there is an undocumented DHCP option code hidden in UniFi's firmware which allows passing the full URL to the inform endpoint. The standard IP-based provisioning uses option 43 code 1 containing an IP address in binary format. But there is also code 2 which takes a full URL in text format. Sadly configuration of these vendor-specific options is very dependant on the used DHCP server, I can only give an example for ISC dhcpd.

option space ubnt;
option ubnt.unifi-address code 1 = ip-address;
# The undocumented URL option
option ubnt.unifi-url code 2 = text;

# Define Ubiquiti vendor class with option space
class "ubnt" {
	match if substring (option vendor-class-identifier , 0, 4) = "ubnt";
	option vendor-class-identifier "ubnt";
	vendor-option-space ubnt;
}

shared-network testing {
	subnet 192.0.2.0 netmask 255.255.255.0 {
		option ubnt.unifi-url "http://unifi.example.com/inform";
        # ...
	}
}

With this configured, all unconfigured UniFi APs in the network will send an adoption request to the given inform endpoint.

Since this option is undocumented by Ubiquiti, it could theoretically go away at any time, but it has been there for at least a few years and three major firmware revisions (4, 5 and 6) so it seems like Ubiquiti has no interest in removing it.

The ASRock DeskMini X300 is a very affordable barebone computer for everything that doesn't require more than 8C/16T or a necessitates an external GPU. Equipped with a 5700G, 32 GiB of RAM and a 512GiB of fast NVMe flash they are fairly powerful machines at a very affordable price point (~800$) while also being space-efficient at around 2l volume, power-efficient (measured at around 7W in idle on Linux) and quiet (at least with a Noctua L9a fitted).

They do however have one giant flaw: Standby (ACPI S3) is just straight-up unavailable out-of-the-box with CPUs starting from the 4000 series. Windows won't even show you the button, Linux attempts to use a S2idle which doesn't do much because the hardware has no S2idle controllers available. It turns the screen off and lowers power consumption by around 1W, but that's about it. The power LED and fan keep running. This is now documented by ASRock, but only on their detailed specifications tab and literally on the last row.

When I got my first batch of them I thought about sending them back because Desktop PCs without ACPI S3 aren't really useable. But first I wanted to try and see if I could enable ACPI S3 on these boxes. Because fundamentally there really isn't much on there that could screw with S3. The X300 chipset isn't really a chipset, but pretty much denotes the lack of a chipset. The only portion of it that's actually physical is a marker chip telling the CPU/SoC that there is no actual chipset present. So basically the only things that are even involved are the CPU/SoC itself, memory, the SuperIO chip and the firmware/BIOS. But all of these components individually support S3, at least if you're not running the CPU with TSME (Transparent System Memory Encryption) enabled. And since the sister platform Jupiter X300 (also by ASRock) supports S3 it's extremely unlikely that the hardware physically can't do it. The reason for the issue is in the firmware/BIOS.

What's interesting about the issue is that the operating systems are aware of the fact that S3 is unsupported. This means that the firmware explicitly chose to pass an ACPI table with S3 disabled. So I dumped the ACPI table responsible for most of the original ACPI power handling, the Differentiated System Description Table (DSDT). I tried decompiling the table with acpica, but the decompiler refused because the table contained broken definitions. I patched the decompiler to accept the broken declarations and saw something very interesting

    Name (_S0, Package (0x04)  // _S0_: S0 System State
    {
        Zero, 
        Zero, 
        Zero, 
        Zero
    })
    Name (XS3, Package (0x04)
    {
        0x03, 
        Zero, 
        Zero, 
        Zero
    })
    Name (_S4, Package (0x04)  // _S4_: S4 System State
    {
        0x04, 
        Zero, 
        Zero, 
        Zero
    })
    Name (_S5, Package (0x04)  // _S5_: S5 System State
    {
        0x05, 
        Zero, 
        Zero, 
        Zero
    })

For the S0 (powered on), S4 (suspend-to-disk) and S5 (powered off) power states, there is a valid entry (see the ACPI sleeping state docs), but for S3 there is an X in front of it, which is not a defined ACPI resource name. Because this platform supports S3 on older CPUs and the entry still exists I wrote a patch which replaced the XS3 resource name with _S3 which is the proper one and recompiled the DSDT table using acpica's iasl. But ACPI tables are provided by firmware, so just having a fixed one doesn't really get you far. Luckily Linux provides a facility to overwrite firmware ACPI tables using a special type of CPIO archive / initramfs. The compiled ACPI table needs to be located in the CPIO archive at kernel/firmware/acpi/dsdt.aml. Then this CPIO archive needs to be passed to your bootloader of choice before the normal initramfs. There's one more caviat which bit me the first time I tried this: ACPI tables have a version number. Linux will only load the override table if its version is higher than the one provided by firmware. Another small patch to the DefinitionBlock of the table source later, Linux loaded my modified DSDT table:

ACPI: Table Upgrade: override [DSDT-ALASKA-  A M I ]
ACPI: DSDT 0x00000000BB235000 Physical table override, new table: 0x00000000BD632000

And unsurprisingly it now showed S3 as supported:

ACPI: PM: (supports S0 S3 S4 S5)

Now I needed to actually test it since I only forced ACPI to expose the capability and didn't yet fix any possible bugs with S3. But to my surprise just running systemctl suspend suspended the machine and pressing the power button resumed it again without any obvious bugs. A stress test with 100 sleep/wake cycles didn't reveal any issues at least on my Cezanne-based Ryzen 5700G and with TSME disabled in firmware. Success!

I also looked into modifying the firmware itself to correct the ACPI table there but I didn't find the responsible EFI module. Modern EFI firmware is one massive pile of code and at least for AMD firmware there isn't much public research/documentation available yet. So for now I'm injecting my fix via Linux's ACPI table override feature. If someone wants to have a go at it, the firmware is here.

A few notes on the constraints of this hack:

1. This is unsupported by both the vendor (ASRock) and me. I am not responsible for dead hardware or eaten cats. And depending on your jurisdiction it might void your warranty.
2. Any change to firmware settings can change the underlying ACPI tables. Since the patched one is not taken from the firmware you then have an inconsistent set of ACPI tables loaded which can lead to unpredictable behavior and in extreme cases even hardware damage. The only "safe" way to do this is to never change firmware settings or update the firmware after you've injected the patch. Otherwise you need to remove the hack first, change the settings and then redo the whole procedure.
3. This is only confirmed to work on Cezanne-based APUs (so the Ryzen 5x00G series) and Renoir-based APUs (Ryzen 4x50G, thanks Z3NOX) with TSME disabled. It might very well not work with other APUs.
4. It only works if your operating system supports overriding ACPI tables. I don't know how to do that on Windows and have never tested it.

If you have one of those boxes and you want to apply my fix, there is a repo at https://github.com/lorenz/asrock-x300-s3-fix which contains instructions on how to use it.

I wanted to share a small trick for using Go 1.11+ modules with GitLab repos. If you just try to import another repo and run a build or test using GitLab CI, this happens:

fatal: could not read Username for 'https://git.dolansoft.org': terminal prompts disabled

Go (calling Git) complains because it doesn't know how to authenticate itself. GitLab has a cool feature as of 9.0 which grants the GitLab CI token the same access rights as the user that pushed the commit. Now we just need to make Git use that. We cannot pass these credentials in the URL since the call is controlled by Go. But there exists a Git feature which allows us to replace the URL by one of our choosing, including one that contains credentials. The end result looks something like this:

test:
  stage: test
  image: golang:1.12
  script:
    - git config --global url."https://gitlab-ci-token:$CI_BUILD_TOKEN@git.dolansoft.org/".insteadOf "https://git.dolansoft.org/"
    - go test

If you have multiple build steps, you can also put that command into before_script.

Recently I needed to expand my disk storage in my server. I previously had 2x2TB of old WD Green disks in a BTRFS RAID1. Now I upgraded to 8x3TB and a dedicated SAS2008-based controller.

As a filesystem I opted for ZFS because I wanted parity-based RAID and Btrfs's implementation is considered broken at the moment. Because my server (just like all servers at DolanSoft) run on CoreOS I needed to compile ZFS for CoreOS.

That is not easily done though, because there is no kernel module compilation environment in CoreOS and because it is an immutable operating system, there is not even the possibility of installing one.

Edit 01/18: I now published torcx-zfs which is a much cleaner and quicker way of installing ZFS on CoreOS

After a lot of digging, I found this mail by a CoreOS dev which linked to a container that is automatically built for every CoreOS version and has the needed tools to compile the ZFS kernel module and userspace tools.

So I pulled the right container for my CoreOS version (stable)

wget http://stable.release.core-os.net/amd64-usr/current/coreos_developer_container.bin.bz2
bunzip2 coreos_developer_container.bin.bz2

and started the resulting container using sudo systemd-nspawn -i coreos_developer_container.bin --share-system.

The mail above also had instructions on how to prepare that container for kernel module development, which I'm repeating here:

emerge-gitclone
emerge -gKav coreos-sources
cd /usr/src/linux
zcat /proc/config.gz >.config
make modules_prepare

After that, we're ready to build SPI and ZFS:

wget -O - https://github.com/zfsonlinux/zfs/releases/download/zfs-0.6.5.7/zfs-0.6.5.7.tar.gz | tar -xzf -
wget -O - https://github.com/zfsonlinux/spl/archive/spl-0.6.5.7.tar.gz | tar -xzf -
cd spl && make && make install
cd zfs && make && make install

The resulting coreos_developer_container.bin can now be used anywhere to install the ZFS kernel module and userspace.

sudo systemd-nspawn -i coreos_developer_container.bin --bind /:/target --capability=CAP_SYS_MODULE --share-system
modprobe zfs
cp /usr/local/sbin/* /target/opt/bin/
cp -r /usr/local/lib64 /target/usr/share/oem/
chown -R 755 /target/usr/share/oem

Now we need to run ldconfig -v to update the library path and restart the session.

We can now finally create a ZFS pool:

sudo zpool create data raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sdf

Have fun with your new ZFS!