Enable clang COFF native build toolchain in edk2

We've enabled the windows version clang tool-chain through the CLANG38 in edk2, which is based on ld.lld and has GCC style build options. The new CLANG38+ld.lld can pass edk2 build in both windows and Linux now. The enabling details: https://github.com/shijunjing/edk2/wiki/LLVM-LLD-linker-enabling-in-edk2-build

The Clang38 is good, but it need two link steps to generate Uefi executable binary: EFL binary firstly, then convert EFL to COFF with GenFw tool. Because the llvm linker offer a native COFF linker, lld-link, which can directly produce the COFF executable DLL binary for efi module. We don't need generate EFL binary firstly then convert EFL to COFF actually.

The llvm linker owner suggest me to use clang-cl (clang MSVC interface compiler) + lld-link (llvm COFF linker) solution to directly produce the COFF native executable for Uefi in both Linux and Windows, which can avoid the ELF complex relocation convent and make the Linux build much easier. The “clang-cl + lld-link” solution support both PDB and dwarf debug info and looks can cover all Uefi build and debug requirements. See the background detail here: http://lists.llvm.org/pipermail/llvm-dev/2019-January/129749.html

Clang-CL is the driver program for clang that attempts to be compatible with MSVC’s cl.exe. https://clang.llvm.org/docs/MSVCCompatibility.html

20190213

~/wksp_efi/lgao4/edk2$ export CLANGCL8_BIN= ~/llvm/releaseinstall/bin/
~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGCL8 -b DEBUG -m MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe.inf
cd ~/wksp_efi/edk2
~/wksp_efi/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -b DEBUG -a IA32 -a X64 -t CLANG38
~/wksp_efi/edk2$ cp ~/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/UsbBusDxe.efi Build/rootfs/
~/wksp_efi/edk2$ qemu-system-x86_64 -m 5120 -smp 1 -bios ~/wksp_efi/edk2/Build/Ovmf3264/DEBUG_CLANG38/FV/OVMF.fd  -machine q35 -serial mon:stdio -display none --net none -hda fat:Build/rootfs/
boot into uefi shell
Shell> reconnect -r
Shell> map -r
fs0:
FS0:\> load UsbBusDxe.efi

20190214

Meet a problem that the clang-cl see the ssdt.i file as "previously preprocessed input", and always skip its prepossess. This causes the output file, ssdt.iii, be a empty file.

~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGCL8 -b DEBUG -m OvmfPkg/AcpiTables/AcpiTables.inf
Building ... /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables/AcpiTables.inf [X64]
Trim --asl-file -o /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.i -i /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/inc.lst /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables/Ssdt.asl
"/home/jshi19/llvm/releaseinstall/bin/clang-cl" /nologo /E /C /FIAutoGen.h -I/home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables -I/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/DEBUG -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include/X64 -I/home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg -I/home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/Include /I/home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.i > /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.iii
clang-8: warning: /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.i: previously preprocessed input [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-C' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-include AutoGen.h' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/DEBUG' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/MdePkg' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include' [-Wunused-command-line-argument]
Trim --source-code -l -o /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.iiii /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.iii
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include/X64' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/Include' [-Wunused-command-line-argument]
clang-8: warning: argument unused during compilation: '-I /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/AcpiTables' [-Wunused-command-line-argument]
"iasl"  -p/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.aml /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/OvmfPkg/AcpiTables/AcpiTables/OUTPUT/./Ssdt.iiii

Error    6126 -  Input file does not appear to be an ASL or data table source file

This problem is solved by add /TC in ASLPP flag as below to force treat all source files as C

*_CLANGCL8_*_ASLPP_FLAGS            = /nologo /E /C /TC /FIAutoGen.h

Meet another problem that the clang-cl preprocess output is different from MSVC, and VfrCompile cannot support it.

The clang-cl and MSVC prepreocess difference:

msvc:     #line 1 "d:\\project\\edk2git\\workspace\\edk2\\MdeModulePkg\\Library\\BootManagerUiLib\\BootManagerVfr.Vfr"
clang-cl: # 1 "/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Library/BootManagerUiLib/BootManagerVfr.Vfr"
clang-cl: # 1 "<built-in>" 1

You can see there are two differences. The first is the #line and #. The second is clang-cl has one more number "1" at the end of some # strings. And VfrCompile doesn't support it and report error as below:

"/home/jshi19/llvm/releaseinstall/bin/clang-cl" /nologo /E /TC /DVFRCOMPILE /FIBootManagerUiLibStrDefs.h -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Library/BootManagerUiLib -I/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/DEBUG -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include/X64 -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Include /home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Library/BootManagerUiLib/BootManagerVfr.Vfr > /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/OUTPUT/BootManagerVfr.i
"VfrCompile" -l -n --string-db /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/OUTPUT/BootManagerUiLibStrDefs.hpk --output-directory /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/DEBUG/. /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/OUTPUT/BootManagerVfr.i
unrecognized input 'invalid token'
unrecognized input 'invalid token'
VfrCompile...
/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib/OUTPUT/BootManagerVfr.i(1): ERROR 12288: 1
        : unexpected token

unrecognized input 'invalid token'
unrecognized input 'invalid token'
unrecognized input 'invalid token'
unrecognized input 'invalid token'
unrecognized input 'invalid token'
unrecognized input 'invalid token'

If change the above two clang-cl strings as below, the VfrCompile pass build.

clang-cl: #line 1 "/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Library/BootManagerUiLib/BootManagerVfr.Vfr"
clang-cl: #line 1 "<built-in>"

20190215

Vfrcompiler cannot recognize the comment format of clang-cl preprocess output. Talked with Vfrcomipler owner, the Vfrcompiler parse the token through edk2\BaseTools\Source\C\VfrCompile\Pccts\antlr tool. Did some quick try to change the antlr tool (e.g. antlr.g), but still failed to let Vfrcompiler recognize the comment format of clang-cl preprocess. Add a work around in dsc and fdf as below to skip the three build fail Vfr modules. Will back to investigate a total solution after solving other issues.

+#  FileExplorerLib|MdeModulePkg/Library/FileExplorerLib/FileExplorerLib.inf

+#  MdeModulePkg/Application/UiApp/UiApp.inf {
+#    <LibraryClasses>
+#      NULL|MdeModulePkg/Library/DeviceManagerUiLib/DeviceManagerUiLib.inf
+#      NULL|MdeModulePkg/Library/BootManagerUiLib/BootManagerUiLib.inf
+#      NULL|MdeModulePkg/Library/BootMaintenanceManagerUiLib/BootMaintenanceManagerUiLib.inf
+#!ifdef $(CSM_ENABLE)
+#      NULL|IntelFrameworkModulePkg/Library/LegacyBootManagerLib/LegacyBootManagerLib.inf
+#      NULL|IntelFrameworkModulePkg/Library/LegacyBootMaintUiLib/LegacyBootMaintUiLib.inf
+#!endif
+#  }

+#  MdeModulePkg/Universal/Disk/RamDiskDxe/RamDiskDxe.inf

lld-link report many undefined symbol when build CpuMpPei module.

$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGCL8 -b DEBUG -m UefiCpuPkg/CpuMpPei/CpuMpPei.inf
"/home/jshi19/llvm/releaseinstall/bin/lld-link" /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/IA32/UefiCpuPkg/CpuMpPei/CpuMpPei/DEBUG/CpuMpPei.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHINE:X86 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER: /DEBUG   @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/IA32/UefiCpuPkg/CpuMpPei/CpuMpPei/OUTPUT/static_library_files.lst
lld-link: error: undefined symbol: _AsmReadEflags
>>> referenced by /home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Library/PeimEntryPoint/PeimEntryPoint.c:52
>>>               lto.tmp:(__ModuleEntryPoint)
>>> referenced by /home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Library/PeimEntryPoint/PeimEntryPoint.c:52
>>>               lto.tmp:(__ModuleEntryPoint)
>>> referenced by /home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Library/PeimEntryPoint/PeimEntryPoint.c:52
>>>               lto.tmp:(__ModuleEntryPoint)

20190218

The CpuMpPei module build failure with clang-cl This issue looks a common issue for all 32bit module build, even the UsbBusDxe module on IA32:

$ build -p OvmfPkg/OvmfPkgIa32.dsc -a IA32 -t CLANGCL8 -b DEBUG -m MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe.inf
lld-link: error: undefined symbol: _IoWriteFifo8
>>> referenced by /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/Library/PlatformDebugLibIoPort/DebugLib.c:82
>>>               lto.tmp:(_DebugPrint)
>>> referenced by /home/jshi19/wksp_efi/lgao4/edk2/OvmfPkg/Library/PlatformDebugLibIoPort/DebugLib.c:128
>>>               lto.tmp:(_DebugAssert)

All the undefined symbols are the nasm assembly functions. E.g. the missing functions:

IoReadFifo8: edk2\MdePkg\Library\BaseIoLibIntrinsic\Ia32\IoFifoSev.nasm
InternalMemSetMem: edk2\MdePkg\Library\BaseMemoryLibRepStr\Ia32\SetMem.nasm

It is easy to check these "undefined symbol" real source code definitions in MSVS build .map file, e.g. UsbBusDxe.map

edk2\Build\Ovmf3264\DEBUG_VS2015x86\X64\MdeModulePkg\Bus\Usb\UsbBusDxe\UsbBusDxe\OUTPUT\UsbBusDxe.map
 0001:00000000       InternalMemCopyMem         00000000000002c0     BaseMemoryLibRepStr:CopyMem.obj
 0001:00000040       InternalMemZeroMem         0000000000000300     BaseMemoryLibRepStr:ZeroMem.obj
 0001:00000060       InternalMemCompareMem      0000000000000320     BaseMemoryLibRepStr:CompareMem.obj
 0001:00000080       InternalMemSetMem          0000000000000340     BaseMemoryLibRepStr:SetMem.obj
 0001:000000a0       InternalMemSetMem64        0000000000000360     BaseMemoryLibRepStr:SetMem64.obj
 0001:000000c0       InternalMemSetMem32        0000000000000380     BaseMemoryLibRepStr:SetMem32.obj
 0001:000000e0       SevNoRepIo                 00000000000003a0     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:00000120       IoReadFifo8                00000000000003e0     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:00000142       IoReadFifo16               0000000000000402     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:00000168       IoReadFifo32               0000000000000428     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:0000018b       IoWriteFifo8               000000000000044b     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:000001ad       IoWriteFifo16              000000000000046d     BaseIoLibIntrinsicSev:IoFifoSev.obj
 0001:000001d3       IoWriteFifo32              0000000000000493     BaseIoLibIntrinsicSev:IoFifoSev.obj

OK, so it is a problem of 32bits nasm functions cannot be linked by lld-link. But the 64bits nasm functions works well with lld-link. It is weird.

Below is a simplified example to reproduce the problem. There are two source code files: main.c and foo.nasm

~/llvm/wrongcode/lld-link$ cat foo.nasm
    SECTION .text
global Foo
Foo:
    push esi
    push edi
    pop edi
    pop esi
    ret

~/llvm/wrongcode/lld-link$ cat main.c
void
Foo (void);
int main()
{
  Foo();
  return 0;
}

32bits linking fails with error of Foo assembly function is a undefined symbol:

* ~/llvm/wrongcode/lld-link$ nasm foo.nasm -Ox -f win32 -g -o foo.obj
* ~/llvm/wrongcode/lld-link$ ~/llvm/releaseinstall/bin/clang-cl main.c  /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /EHs-c- /GR- /GF /Gy /Zi /Gw -m32
* ~/llvm/wrongcode/lld-link$ ~/llvm/releaseinstall/bin/lld-link main.obj foo.obj /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /DLL /ENTRY:main /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER: /DEBUG:GHASH /MACHINE:X86 
lld-link: error: undefined symbol: _Foo
>>> referenced by /home/jshi19/llvm/wrongcode/lld-link/main.c:6
>>>               main.obj:(_main)

64bits linking is successful:

* ~/llvm/wrongcode/lld-link$ nasm foo.nasm -Ox -f win64 -g -o foo.obj 
* ~/llvm/wrongcode/lld-link$ ~/llvm/releaseinstall/bin/clang-cl main.c /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2s /EHs-c- /GR- /GF /Gy /Zi /Gw -m64
* ~/llvm/wrongcode/lld-link$ ~/llvm/releaseinstall/bin/lld-link main.obj foo.obj /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /DLL /ENTRY:main /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER: /DEBUG:GHASH /Machine:X64

Have written to the lld-link owner for the suggestion: http://lists.llvm.org/pipermail/llvm-dev/2019-February/130282.html

Suggestion from the community (http://lists.llvm.org/pipermail/llvm-dev/2019-February/130285.html) that my preprocessor not set right target. After double check my clangcl38 option, I find I missed below two important TARGET option.

DEFINE CLANGCL8_IA32_TARGET          = --target=i686-pc-win32-coff
DEFINE CLANGCL8_X64_TARGET           = --target=x86_64-pc-win32-coff

And I update the PP, VFRPP, APP, ASLCC, ASLPP and CC with the correct Arch TARGET macro. Below is some IA32 example:

* *_CLANGCL8_IA32_PP_FLAGS            = DEF(CLANGCL8_IA32_TARGET) /nologo /E /TC /FIAutoGen.h -m32
* *_CLANGCL8_IA32_VFRPP_FLAGS         = DEF(CLANGCL8_IA32_TARGET) /nologo /E /TC /DVFRCOMPILE /FI$(MODULE_NAME)StrDefs.h
* *_CLANGCL8_IA32_APP_FLAGS           = DEF(CLANGCL8_IA32_TARGET) /nologo /E /TC
* *_CLANGCL8_IA32_ASLCC_FLAGS         = DEF(CLANGCL8_IA32_TARGET) DEF(MSFT_ASLCC_FLAGS)
* *_CLANGCL8_IA32_ASLPP_FLAGS         = DEF(CLANGCL8_IA32_TARGET) /nologo /E /C /TC /FIAutoGen.h
* DEBUG_CLANGCL8_IA32_CC_FLAGS        = DEF(CLANGCL8_IA32_TARGET) /nologo -m32 /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gw -Wno-error=unused-parameter -Wno-error=sign-compare -Wno-error=missing-field-initializers -flto

After the above update, the undefined symbol linker error disappear. But I meet a new problem: the linker crash!

The lld-link crash when linking the 32bits module

~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32.dsc -a IA32 -t CLANGCL8 -b DEBUG -m MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe.inf
"/home/jshi19/llvm/releaseinstall/bin/lld-link" /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/DEBUG/UsbBusDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHINE:X86 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER: /lldmap /DEBUG   @/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/OUTPUT/static_library_files.lst
lld-link: /home/jshi19/llvm/llvm/lib/DebugInfo/CodeView/MergingTypeTableBuilder.cpp:98: llvm::codeview::TypeIndex llvm::codeview::MergingTypeTableBuilder::insertRecordAs(llvm::hash_code, llvm::ArrayRef<unsigned char>&): Assertion `Record.size() % 4 == 0 && "Record is not aligned to 4 bytes!"' failed.
Stack dump:
0.      Program arguments: /home/jshi19/llvm/releaseinstall/bin/lld-link /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/DEBUG/UsbBusDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHINE:X86 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER: /lldmap /DEBUG @/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/OUTPUT/static_library_files.lst
 #0 0x00005583481d246a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x26946a)
 #1 0x00005583481d02e4 llvm::sys::RunSignalHandlers() (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2672e4)
 #2 0x00005583481d0422 SignalHandler(int) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x267422)
 #3 0x00007f961a4ac890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #4 0x00007f9618fa0e97 gsignal /build/glibc-OTsEL5/glibc-2.27/signal/../sysdeps/unix/sysv/linux/raise.c:51:0
 #5 0x00007f9618fa2801 abort /build/glibc-OTsEL5/glibc-2.27/stdlib/abort.c:81:0
 #6 0x00007f9618f9239a __assert_fail_base /build/glibc-OTsEL5/glibc-2.27/assert/assert.c:89:0
 #7 0x00007f9618f92412 (/lib/x86_64-linux-gnu/libc.so.6+0x30412)
 #8 0x000055834987c30f llvm::codeview::MergingTypeTableBuilder::insertRecordAs(llvm::hash_code, llvm::ArrayRef<unsigned char>&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x191330f)
 #9 0x000055834989dfc2 llvm::Error llvm::codeview::forEachCodeViewRecord<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind>, (anonymous namespace)::TypeStreamMerger::remapAllTypes(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> > > const&)::'lambda'(llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> const&)>(llvm::ArrayRef<unsigned char>, (anonymous namespace)::TypeStreamMerger::remapAllTypes(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> > > const&)::'lambda'(llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> const&)) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x1934fc2)
#10 0x000055834989ee8a (anonymous namespace)::TypeStreamMerger::doit(llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> > > const&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x1935e8a)
#11 0x000055834989f54c llvm::codeview::mergeTypeAndIdRecords(llvm::codeview::MergingTypeTableBuilder&, llvm::codeview::MergingTypeTableBuilder&, llvm::SmallVectorImpl<llvm::codeview::TypeIndex>&, llvm::VarStreamArray<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind>, llvm::VarStreamArrayExtractor<llvm::codeview::CVRecord<llvm::codeview::TypeLeafKind> > > const&, llvm::Optional<unsigned int>&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x193654c)
#12 0x0000558348281d36 (anonymous namespace)::PDBLinker::mergeDebugT(lld::coff::ObjFile*, (anonymous namespace)::CVIndexMap*) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x318d36)
#13 0x000055834827f8c3 (anonymous namespace)::PDBLinker::addObjFile(lld::coff::ObjFile*, (anonymous namespace)::CVIndexMap*) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x3168c3)
#14 0x0000558348284442 lld::coff::createPDB(lld::coff::SymbolTable*, llvm::ArrayRef<lld::coff::OutputSection*>, llvm::ArrayRef<unsigned char>, llvm::codeview::DebugInfo*) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x31b442)
#15 0x0000558348258442 lld::coff::writeResult() (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2ef442)
#16 0x0000558348218487 lld::coff::LinkerDriver::link(llvm::ArrayRef<char const*>) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2af487)
#17 0x000055834821b888 lld::coff::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2b2888)
#18 0x00005583481597b0 main (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x1f07b0)
#19 0x00007f9618f83b97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
#20 0x00005583481b88fa _start (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x24f8fa)
Aborted (core dumped)
GNUmakefile:385: recipe for target '/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/DEBUG/UsbBusDxe.dll' failed
make: *** [/home/jshi19/wksp_efi/lgao4/edk2/Build/OvmfIa32/DEBUG_CLANGCL8/IA32/MdeModulePkg/Bus/Usb/UsbBusDxe/UsbBusDxe/DEBUG/UsbBusDxe.dll] Error 134

20190222

Fix the issue that Vfrcompiler cannot recognize the comment format of clang-cl preprocess output. Liming suggest to use /EP instead of /P in the *_VFRPP_FLAGS as below to disable the linemarker comments output in the prepreocess result. The linemarker comment format is different in clang-cl and MSVC. Because Edk2 VfrCompile don't need these linemarker comment. So we can just disable the linemarker comment for all.

BaseTools/Conf/tools_def.template
-*_CLANGCL8_IA32_VFRPP_FLAGS         = DEF(CLANGCL8_IA32_TARGET) /nologo /E /TC /DVFRCOMPILE /FI$(MODULE_NAME)StrDefs.h
+*_CLANGCL8_IA32_VFRPP_FLAGS         = DEF(CLANGCL8_IA32_TARGET) /nologo /EP /TC /DVFRCOMPILE /FI$(MODULE_NAME)StrDefs.h

20190225

Build the SnpDxe module cause the lld-link crash as below. But if remove the /O1b2s in CC_FLAGS, build pass. And NOOPT build also pass. It looks like a lld-link linker bug.

~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGCL8 -b DEBUG -m MdeModulePkg/Universal/Network/SnpDxe/SnpDxe.inf
"/home/jshi19/llvm/releaseinstall/bin/lld-link" /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/DEBUG/SnpDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /Machine:X64 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /lldmap /DEBUG:GHASH   @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/static_library_files.lst
Stack dump:
0.      Program arguments: /home/jshi19/llvm/releaseinstall/bin/lld-link /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/DEBUG/SnpDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /Machine:X64 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /lldmap /DEBUG:GHASH @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/static_library_files.lst
 #0 0x00005561c887746a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x26946a)
 #1 0x00005561c88752e4 llvm::sys::RunSignalHandlers() (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2672e4)
 #2 0x00005561c8875422 SignalHandler(int) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x267422)
 #3 0x00007f703ef2a890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #4 0x00005561c88dc9b5 lld::coff::markLive(llvm::ArrayRef<lld::coff::Chunk*>) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2ce9b5)
 #5 0x00005561c88c0151 lld::coff::LinkerDriver::link(llvm::ArrayRef<char const*>) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2b2151)
 #6 0x00005561c88c0888 lld::coff::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2b2888)
 #7 0x00005561c87fe7b0 main (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x1f07b0)
 #8 0x00007f703da01b97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
 #9 0x00005561c885d8fa _start (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x24f8fa)
Segmentation fault (core dumped)
make: *** [/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/DEBUG/SnpDxe.dll] Error 139

20190226

For the issue of lld-link crash when build SnpDxe module, I suspect it is a lld-link bug, and I'm trying to narrow down the SnpDxe code, and figure out a simpler and reproducible code snippet.

change file content as below:

* /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/object_files.lst:

/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/MemoryIntrinsics.obj

* /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/static_library_files.lst:
/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/BaseLib.lib
/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/SnpDxe.lib

* /home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Universal/Network/SnpDxe/Start.c:

typedef struct {
  struct _MAP_LIST {
    unsigned long long    VirtualAddress;
    void                  *MapCookie;
  } MapList[100];

} _SNP_DRIVER;


int
_SnpUndi32Start (
  _SNP_DRIVER  *Snp
  )
{
  int       Index;

  for (Index = 0; Index < 100; Index++) {
    Snp->MapList[Index].VirtualAddress = 0;
    Snp->MapList[Index].MapCookie      = 0;
  }

  return 0;
}

Then run build command after

~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANGCL8 -b DEBUG -m MdeModulePkg/Universal/Network/SnpDxe/SnpDxe.inf

rm /home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/BaseLib.lib
"/home/jshi19/llvm/releaseinstall/bin/llvm-lib" /NOLOGO /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/BaseLib.lib @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdePkg/Library/BaseLib/BaseLib/OUTPUT/object_files.lst
"/home/jshi19/llvm/releaseinstall/bin/clang-cl" /Fo/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/./Start.obj /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Z7 -Wno-error=unused-parameter -Wno-error=sign-compare -Wno-error=missing-field-initializers -Wno-tautological-pointer-compare -Wno-microsoft-enum-forward-reference -gcodeview --target=x86_64-pc-win32-coff -m64 /O1b2s /Gw -flto /D DISABLE_NEW_DEPRECATED_INTERFACES -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Universal/Network/SnpDxe -I/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/DEBUG -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include -I/home/jshi19/wksp_efi/lgao4/edk2/MdePkg/Include/X64 -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg -I/home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Include /home/jshi19/wksp_efi/lgao4/edk2/MdeModulePkg/Universal/Network/SnpDxe/Start.c
"/home/jshi19/llvm/releaseinstall/bin/llvm-lib" /NOLOGO /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/SnpDxe.lib @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/object_files.lst
"/home/jshi19/llvm/releaseinstall/bin/lld-link" /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/DEBUG/SnpDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /DLL /ENTRY:_SnpUndi32Start /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /lldmap /Machine:X64 /DEBUG:GHASH   @/home/jshi19/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANGCL8/X64/MdeModulePkg/Universal/Network/SnpDxe/SnpDxe/OUTPUT/static_library_files.lst

Need to continue simplify above files.

Confirm current CLANGCL8 tool chain can pass build with NOOPT and boot into shell on all three OVMF platforms. Below is the testing steps. But the DEBUG build still has the lld-link crash issue.

git clone https://github.com/lgao4/edk2.git
C:\steven\lgao4\edk2>git checkout Clang
C:\steven\lgao4\edk2>set CLANGCL8_BIN=C:\steven\LLVM\LLVM_InstallRel\bin\
C:\steven\lgao4\edk2>edksetup.bat Reconfig
C:\steven\lgao4\edk2>build -t CLANGCL8 -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b NOOPT
C:\steven\lgao4\edk2>"C:\Program Files\qemu\qemu-system-x86_64.exe"  -pflash Build\Ovmf3264\NOOPT_CLANGCL8\FV\OVMF.fd -debugcon file:serial.log -global isa-debugcon.iobase=0x402
C:\steven\lgao4\edk2>build -t CLANGCL8 -p OvmfPkg\OvmfPkgX64.dsc -a X64 -b NOOPT
C:\steven\lgao4\edk2>"C:\Program Files\qemu\qemu-system-x86_64.exe"  -pflash Build\Ovmfx64\NOOPT_CLANGCL8\FV\OVMF.fd -debugcon file:serial.log -global isa-debugcon.iobase=0x402
C:\steven\lgao4\edk2>build -t CLANGCL8 -p OvmfPkg\OvmfPkgIa32.dsc -a IA32 -b NOOPT
C:\steven\lgao4\edk2>"C:\Program Files\qemu\qemu-system-x86_64.exe"  -pflash Build\OvmfIa32\NOOPT_CLANGCL8\FV\OVMF.fd -debugcon file:serial.log -global isa-debugcon.iobase=0x402

20190227

For the issue of lld-link crash when build SnpDxe module, figure out a simplified reproduce example and sent it to the lld-link owner for advice: http://lists.llvm.org/pipermail/llvm-dev/2019-February/130593.html

$ cat main.c
typedef struct {
  struct MAP_LIST {
    unsigned long long  VirtualAddress;
    void              *MapCookie;
  } MapList[100];
} SNP_DRIVER;
SNP_DRIVER snp_instance;
int main()
{
  SNP_DRIVER  *Snp;
  Snp = &snp_instance;
  for (int Index = 0; Index < 100; Index++) {
    Snp->MapList[Index].VirtualAddress = 0;
    Snp->MapList[Index].MapCookie      = 0;
  }
  return 0;
}

$ cat intrinsics.c
void * memset (void *dest, int ch, size_t count)
{
  volatile char  *Pointer;
  Pointer = (char *)dest;
  while (count-- != 0) {
    *(Pointer++) = (char)ch;
  }
  return dest;
}

$ "/home/jshi19/llvm/releaseinstall/bin/clang-cl" /Fomain.obj /c --target=x86_64-pc-win32-coff -m64 /O1b2s -flto main.c
$ "/home/jshi19/llvm/releaseinstall/bin/clang-cl" /Fointrinsics.obj /c --target=x86_64-pc-win32-coff -m64 /O1b2s -flto intrinsics.c
$ "/home/jshi19/llvm/releaseinstall/bin/llvm-lib" /OUT:intrinsics.lib intrinsics.obj
$ "/home/jshi19/llvm/releaseinstall/bin/lld-link" /NODEFAULTLIB /ENTRY:main  main.obj intrinsics.lib
Stack dump:
0.      Program arguments: /home/jshi19/llvm/releaseinstall/bin/lld-link /NODEFAULTLIB /ENTRY:main main.obj intrinsics.lib
#0 0x0000559a34ba5e4a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x272e4a)
#1 0x0000559a34ba3d14 llvm::sys::RunSignalHandlers() (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x270d14)
#2 0x0000559a34ba3e52 SignalHandler(int) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x270e52)
#3 0x00007f896c8ea890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
#4 0x0000559a34c11325 lld::coff::markLive(llvm::ArrayRef<lld::coff::Chunk*>) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2de325)
#5 0x0000559a34bf0220 lld::coff::LinkerDriver::link(llvm::ArrayRef<char const*>) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2bd220)
#6 0x0000559a34bf0478 lld::coff::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x2bd478)
#7 0x0000559a34b2a300 main (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x1f7300)
#8 0x00007f896b3c1b97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
#9 0x0000559a34b8c27a _start (/home/jshi19/llvm/releaseinstall/bin/lld-link+0x25927a)
Segmentation fault (core dumped)

$ "/home/jshi19/llvm/releaseinstall/bin/clang-cl" --version
clang version 9.0.0 (https://github.com/llvm-mirror/clang.git 1f02068469ff18f5fc5728cafe9d96ee5f66c5b9) (https://github.com/llvm-project/llvm.git 330395ea4fce35b019b33797ff751be029a1f866)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: /home/jshi19/llvm/releaseinstall/bin

$ "/home/jshi19/llvm/releaseinstall/bin/lld-link" --version
LLD 9.0.0 (https://github.com/llvm-mirror/lld.git aa7adc0ec804b689771f11d52e39f83a16378f5f)

20190228

For the issue of lld-link crash when build SnpDxe module, the lld-link owner confirm it is a linker bug and will fix it: "http://lists.llvm.org/pipermail/llvm-dev/2019-February/130632.html":http://lists.llvm.org/pipermail/llvm-dev/2019-February/130632.html

20190521

Andrew's suggestion: https://edk2.groups.io/g/devel/message/41091

20190704

Build clang, lld and other llvm tools in linux:

jshi19@ub2-uefi-b01:~/llvm$ git clone https://github.com/llvm/llvm-project.git
jshi19@ub2-uefi-b01:~/llvm$ cd llvm-project/releasebuild/
jshi19@ub2-uefi-b01:~/llvm/llvm-project/releasebuild$ cmake ../llvm -G "Unix Makefiles" -DCMAKE_BUILD_TYPE="Release" -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_CXX_COMPILER="/home/jshi19/wksp_efi/distcc/distcc_ln/g++" -DCMAKE_C_COMPILER="/home/jshi19/wksp_efi/distcc/distcc_ln/gcc" -DLLVM_OPTIMIZED_TABLEGEN=ON -DLLVM_BINUTILS_INCDIR=/home/jshi19/llvm/binutils-2.31/include -DLLVM_INSTALL_UTILS=ON -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_CXX_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_INSTALL_PREFIX=../releaseinstall/ -DLLVM_ENABLE_PROJECTS="lld;clang;compiler-rt;clang-tools-extra"
jshi19@ub2-uefi-b01:~/llvm/llvm-project/releasebuild$ make -j 24
jshi19@ub2-uefi-b01:~/llvm/llvm-project/releasebuild$ make install -j 4

Change the section sector algnment from 512 to 4096 and build again:

jshi19@ub2-uefi-b01:~/llvm/llvm-project$ git diff
diff --git a/lld/COFF/Config.h b/lld/COFF/Config.h
@@ -180,7 +180,7 @@ struct Configuration {
   std::string MapFile;

   uint64_t ImageBase = -1;
-  uint64_t FileAlign = 512;
+  uint64_t FileAlign = 4096;
   uint64_t StackReserve = 1024 * 1024;
   uint64_t StackCommit = 4096;
   uint64_t HeapReserve = 1024 * 1024;
jshi19@ub2-uefi-b01:~/llvm/llvm-project/releasebuild$ make -j 24
jshi19@ub2-uefi-b01:~/llvm/llvm-project/releasebuild$ make install -j 4

Build the edk2 ovmf with the latest clang and lld in linux, and boot qemu in linux successfully!

jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ git diff
diff --git a/BaseTools/Conf/tools_def.template b/BaseTools/Conf/tools_def.template
@@ -5089,9 +5089,9 @@ NOOPT_CLANGxVS15_X64_DLINK_FLAGS    = /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:RE
 ####################################################################################
 *_CLANG7WIN_*_*_FAMILY                = GCC
 *_CLANG7WIN_*_*_BUILDRULEFAMILY       = CLANGWIN
-*_CLANG7WIN_*_MAKE_PATH               = nmake
+*_CLANG7WIN_*_MAKE_PATH               = make
 *_CLANG7WIN_*_*_DLL                   = ENV(CLANG7WIN_DLL)
-*_CLANG7WIN_*_ASL_PATH                = DEF(DEFAULT_WIN_ASL_BIN)
+*_CLANG7WIN_*_ASL_PATH                = DEF(UNIX_IASL_BIN)

 *_CLANG7WIN_*_APP_FLAGS               =
 *_CLANG7WIN_*_ASL_FLAGS               = DEF(DEFAULT_WIN_ASL_FLAGS)

jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ source edksetup.sh
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ export CLANGCL8_BIN=~/llvm/llvm-project/releaseinstall/bin/
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ export CLANG7WIN_BIN=~/llvm/llvm-project/releaseinstall/bin/
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ ./edksetup.sh --reconfig
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANG7WIN -DDEBUG_ON_SERIAL_PORT
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ qemu-system-x86_64 -m 5120 -smp 1 -bios ~/wksp_efi/lgao4/edk2/Build/Ovmf3264/DEBUG_CLANG7WIN/FV/OVMF.fd -global e1000.romfile=""  -machine q35 -serial mon:stdio -display none

Build clang, lld and other llvm tools in windows with Ninja + Visual Studio 2017 build:

Open "Developer command Prompt for VS2017"  with administrator permission
C:\steven\LLVM\llvm-project\releasebuild>cmake -GNinja ..\llvm -DCMAKE_BUILD_TYPE="Release" -DLLVM_TARGETS_TO_BUILD="X86" -DCMAKE_VERBOSE_MAKEFILE=ON -DLLVM_OPTIMIZED_TABLEGEN=ON  -DLLVM_INSTALL_UTILS=ON -DLLVM_ENABLE_ASSERTIONS=ON -DCMAKE_C_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_CXX_FLAGS=-DLLVM_ENABLE_DUMP -DCMAKE_INSTALL_PREFIX=..\releaseinstall\ -DLLVM_ENABLE_PROJECTS="lld;clang;compiler-rt;clang-tools-extra"
C:\steven\LLVM\llvm-project\releasebuild>ninja
C:\steven\LLVM\llvm-project\releasebuild>ninja install

Build the edk2 ovmf with the latest clang and lld in windows, and boot qemu in windows successfully!

C:\steven\lgao4\edk2>edksetup.bat ForceRebuild
C:\steven\lgao4\edk2>edksetup.bat Reconfig
C:\steven\lgao4\edk2>set CLANG7WIN_BIN=C:\steven\LLVM\llvm-project\releaseinstall\bin\
C:\steven\lgao4\edk2>build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANG7WIN -DDEBUG_ON_SERIAL_PORT
C:\steven\lgao4\edk2>"C:\Program Files\qemu\qemu-system-x86_64.exe" -m 5120 -smp 1 -bios c:\steven\lgao4\edk2\Build\Ovmf3264\DEBUG_CLANG7WIN\FV\OVMF.fd -global e1000.romfile=""  -machine q35 -serial mon:stdio

20190705

jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link2$ ~/llvm/llvm-project/releaseinstall/bin/clang -c -Oz -flto -target x86_64-unknown-windows main.c -o main.obj
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link2$ ~/llvm/llvm-project/releaseinstall/bin/lld-link /machine:X64 /DLL /ENTRY:mainfunction main.obj /OUT:main.dll
:mainfunction main.obj /OUT:main.dll
lld-link: error: undefined symbol: _fltused
>>> referenced by lto.tmp

jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link2$ ~/llvm/llvm-project/releaseinstall/bin/clang -c -Oz -flto -target i686-unknown-windows main.c -o main.obj
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link2$ ~/llvm/llvm-project/releaseinstall/bin/lld-link /machine:X86 /DLL /ENTRY:mainfunction main.obj /OUT:main.dll
lld-link: error: undefined symbol: __fltused
>>> referenced by lto.tmp

Have asked above issue in llvm dev forum and got feedback: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133656.html. Then figure out a solution as below to add the symbol _fltused.

diff --git a/CryptoPkg/Library/IntrinsicLib/MemoryIntrinsics.c b/CryptoPkg/Library/IntrinsicLib/MemoryIntrinsics.c
@@ -21,7 +21,16 @@ typedef UINTN  size_t;

 /* OpenSSL will use floating point support, and C compiler produces the _fltused
    symbol by default. Simply define this symbol here to satisfy the linker. */
-int _fltused = 1;
+volatile int _fltused = 0;
+RETURN_STATUS
+EFIAPI
+IntrinsicLibConstructor (
+  VOID
+  )
+{
+  _fltused = 0;
+  return RETURN_SUCCESS;
+}

Now the NetworkPkg X64 can build pass, but IA32 build fails with lld crash at TlsDxe.inf:

jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ build -p NetworkPkg/NetworkPkg.dsc -a X64 -t CLANG7WIN
jshi19@ub2-uefi-b01:~/wksp_efi/lgao4/edk2$ build -p NetworkPkg/NetworkPkg.dsc -a IA32 -t CLANG7WIN
Stack dump:
0.      Program arguments: /home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link /OUT:/home/jshi19/wksp_efi/lgao4/edk2/Build/NetworkPkg/DEBUG_CLANG7WIN/IA32/NetworkPkg/TlsDxe/TlsDxe/DEBUG/TlsDxe.dll /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:REF /OPT:ICF=10 /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHINE:X86 /DLL /ENTRY:_ModuleEntryPoint /SUBSYSTEM:EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DEBUG:GHASH @/home/jshi19/wksp_efi/lgao4/edk2/Build/NetworkPkg/DEBUG_CLANG7WIN/IA32/NetworkPkg/TlsDxe/TlsDxe/OUTPUT/static_library_files.lst
 #0 0x000055557fc3468a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x29068a)
 #1 0x000055557fc32364 llvm::sys::RunSignalHandlers() (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x28e364)
 #2 0x000055557fc324a2 SignalHandler(int) (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x28e4a2)
 #3 0x00007fcf50f6a890 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x12890)
 #4 0x000055557fc8c6a5 lld::coff::markLive(llvm::ArrayRef<lld::coff::Chunk*>) (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x2e86a5)
 #5 0x000055557fc68a9b lld::coff::LinkerDriver::link(llvm::ArrayRef<char const*>) (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x2c4a9b)
 #6 0x000055557fc696a8 lld::coff::link(llvm::ArrayRef<char const*>, bool, llvm::raw_ostream&) (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x2c56a8)
 #7 0x000055557fba4b6c main (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x200b6c)
 #8 0x00007fcf4fa41b97 __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:344:0
 #9 0x000055557fc0080a _start (/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link+0x25c80a)
Segmentation fault (core dumped)

20190711

I narrow down the below code snippet from openssl which can reproduce the lld-link crash. I’ve simplified the TlsDxe module and create a new branch: lldcrash_TlsDxe for our further narrow.

unsigned char *ssl_add_serverhello_tlsext_new(unsigned char *buf)
{
    unsigned char *ret = buf;
    const unsigned char cryptopro_ext[36] = {
        0xfd, 0xe8,         /* 65000 */
        0x00, 0x20,         /* 32 bytes length */
        0x30, 0x1e, 0x30, 0x08, 0x06, 0x06, 0x2a, 0x85,
        0x03, 0x02, 0x02, 0x09, 0x30, 0x08, 0x06, 0x06,
        0x2a, 0x85, 0x03, 0x02, 0x02, 0x16, 0x30, 0x08,
        0x06, 0x06, 0x2a, 0x85, 0x03, 0x02, 0x02, 0x17
    };
    CopyMem(ret, cryptopro_ext, sizeof(cryptopro_ext));
    return ret;
}

Please try the below command to build the TlsDxe module in the lldcrash_TlsDxe branch and continue to simplify it and figure out a minimal test case.

$ git checkout lldcrash_TlsDxe
$ build -p NetworkPkg/NetworkPkg.dsc -a IA32 -t CLANG7WIN  -m NetworkPkg/TlsDxe/TlsDxe.inf

20190715

lld owner confirmed above crash issue is a lld coff bug: http://lists.llvm.org/pipermail/llvm-dev/2019-July/133969.html

I've submitted the BZ for this issue: Bug 42626 - lld-link crash when build openssl with LTO https://bugs.llvm.org/show_bug.cgi?id=42626

20190803

Debug the lld-link section alignment option doesn't work issue. Install the readpe command from http://pev.sourceforge.net/ in both linux and windows:

$ sudo apt install pev
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ make
"/home/jshi19/llvm/llvm-project/releaseinstall/bin/clang" -target i686-unknown-windows -fno-builtin -c -o main.obj  main.c
"/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link" /OUT:main.dll /MACHINE:X86 /DLL /ENTRY:main /ALIGN:32 /FILEALIGN:32 main.obj
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ readpe main.dll
DOS Header
    Magic number:                    0x5a4d (MZ)
    Bytes in last page:              120
    Pages in file:                   1
    Relocations:                     0
    Size of header in paragraphs:    4
    Minimum extra paragraphs:        0
    Maximum extra paragraphs:        0
    Initial (relative) SS value:     0
    Initial SP value:                0
    Initial IP value:                0
    Initial (relative) CS value:     0
    Address of relocation table:     0x40
    Overlay number:                  0
    OEM identifier:                  0
    OEM information:                 0
    PE header offset:                0x78
COFF/File header
    Machine:                         0x14c IMAGE_FILE_MACHINE_I386
    Number of sections:              1
    Date/time stamp:                 1564834257 (Sat, 03 Aug 2019 12:10:57 UTC)
    Symbol Table offset:             0
    Number of symbols:               0
    Size of optional header:         0xe0
    Characteristics:                 0x2102
    Characteristics names
                                         IMAGE_FILE_EXECUTABLE_IMAGE
                                         IMAGE_FILE_32BIT_MACHINE
                                         IMAGE_FILE_DLL
Optional/Image header
    Magic number:                    0x10b (PE32)
    Linker major version:            14
    Linker minor version:            0
    Size of .text section:           0x20
    Size of .data section:           0
    Size of .bss section:            0
    Entrypoint:                      0x1000
    Address of .text section:        0x1000
    Address of .data section:        0
    ImageBase:                       0x10000000
    Alignment of sections:           0x1000
    Alignment factor:                0x20
...
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ ll main.dll
-rwxrwxr-x 1 jshi19 jshi19 544 Aug  3 20:10 main.dll*
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ "/home/jshi19/llvm/llvm-project/releaseinstall/bin/lld-link" /OUT:main.dll /MACHINE:X86 /DLL /ENTRY:main /ALIGN:32 main.obj /FILEALIGN:4096
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ ll main.dll
-rwxrwxr-x 1 jshi19 jshi19 8192 Aug  3 20:14 main.dll*
jshi19@ub2-uefi-b01:~/llvm/wrongcode/lld-link4$ readpe main.dll
... ...
    Alignment of sections:           0x1000
    Alignment factor:                0x1000
... ...

OK, we can see current the lld-link support the /FILEALIGN option well and the file alignment can be specified by user option. But the /ALIGN option doesn't work and always be the 4096 . From the lld source code as below, we can see the pe->SectionAlignment is hardcoded as 4096. This is why the /ALIGN option doesn't respect the user's input value.

lld\COFF\Writer.h
  static const int pageSize = 4096;

lld\COFF\Writer.cpp
  pe->SectionAlignment = pageSize;
  pe->FileAlignment = config->fileAlign;

Use the readpe command to see the PE/COFF header info in windows:

C:\steven\pev-0.80-win32>readpe.exe c:\steven\edk2-fork\Build\Ovmf3264\NOOPT_VS2015x86\IA32\PeiCore.efi -H
...
    Alignment of sections:           0x20
    Alignment factor:                0x20
...

Have sent mail to lld owner for this issue: http://lists.llvm.org/pipermail/llvm-dev/2019-August/134351.html

20190805

lld owner confirmed it is a bug and can be enhanced to not hard-coded the /ALIGN option. The owner filed this bug as https://reviews.llvm.org/D65736 and created a patch for me to test. Have verified the patch can fix the /ALIGN option issue.

20190806

Now, the GenFv has a crash problem with the new /ALIGN:32 and /FILEALIGN:32 option. We need debug it and fix it.

$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANG7WIN
… 
Segmentation fault (core dumped)
### ['GenFv', '-a', '/home/jshi19/wksp_efi/lgao4/edk2-2/Build/Ovmf3264/DEBUG_CLANG7WIN/FV/Ffs/PEIFV.inf', '-o', '/home/jshi19/wksp_efi/lgao4/edk2-2/Build/Ovmf3264/DEBUG_CLANG7WIN/FV/PEIFV.Fv', '-i', '/home/jshi19/wksp_efi/lgao4/edk2-2/Build/Ovmf3264/DEBUG_CLANG7WIN/FV/PEIFV.inf']

20190809

The latest lld have solved the BaseOfCode issue with /ALIGN option. All the three OVMF IA32 X64 IA32X64 platforms in ClangRebase2 branch can pass build with latest lld and boot into shell with /ALIGN:32 and /FILEALIGN:32 option together. I’ve added lld link option /FILEALIGN:32 in ClangRebase2 CLANG7WIN toolchain.

20190816

GenFw release build issue is fixed. Now all the three OVMF IA32 X64 IA32X64 platforms, both release and debug, in ClangRebase2 branch can pass build with latest lld and boot into shell!

OK, only last blocking issue left:

Bug 42626 - lld-link crash when build openssl with LTO: https://bugs.llvm.org/show_bug.cgi?id=42626

20190817

Good news!! Google Amy offered me a patch and now the lld can pass the NetworkPkg build. The lld final patch is expected to check-in in next week. Now, both two lld blocking issues as below are solved. All OVMF platforms (RELEASE/DEBUG, IA32, X64, IA32X64) in the branch ClangRebase2 can pass build with CLANG7WIN and boot into shell.

Bug 42626 - lld-link crash when build openssl with LTO https://bugs.llvm.org/show_bug.cgi?id=42626
Bug 42887 - lld/COFF does not handle /align https://bugs.llvm.org/show_bug.cgi?id=42887

20190826

Google Hans, Wennborg confirms the fix of Bug 42626 and Bug 42887 already been merged in LLVM9.0 release! It's great!!

20190902

working on the across-host reproducible build (https://reproducible-builds.org) toolchain enabling which is based on LLVM9 clang + lld. This means to use single toolchain (e.g. LLVM9) can generate exactly same self-contained firmware binary from same source code in different OS (e.g. windows and Linux). There is very big value behind this capablity for firmware. You only need build once then get exactly same bianry for all OS platforms. And the binary itself can be the fingerprint of your toolchain + source code, which is very useful for security and bug reproducing.

But current edk2 AutoGen does not support it. The AutoGen.h and AutoGen.c are generated differently for same module in windows and linux. This problem is common for all toolchains actually, and you can even reproduce this issue through VS2015x86 and GCC5 toolchain. Have submitted a BZ to require enhance it as below:

AutoGen need generate same AutoGen.h and AutoGen.c for same module across different OS https://bugzilla.tianocore.org/show_bug.cgi?id=2147

20190903

Continue the across-host reproducible build work. The SecMain.efi is different because the LzmaDec_DecodeReal2() has a MY_FAST_CALL macro which is expanded to the different tokens on Windows and Linux as below.

Linux:
static int LzmaDec_DecodeReal2(CLzmaDec *p, SizeT limit, const Byte *bufLimit)
Windows:
static int __fastcall LzmaDec_DecodeReal2(CLzmaDec *p, SizeT limit, const Byte *bufLimit)

Liming offer a patch to fix it:

diff --git a/MdeModulePkg/Library/LzmaCustomDecompressLib/Sdk/C/7zTypes.h b/MdeModulePkg/Library/LzmaCustomDecompressLib/Sdk/C/7zTypes.h
@@ -118,7 +118,7 @@ typedef int Bool;
 #define MY_STD_CALL
 #endif
 
-#ifdef _MSC_VER
+#if defined(_MSC_VER) && !defined(__clang__)
 
 #if _MSC_VER >= 1300
 #define MY_NO_INLINE __declspec(noinline)

20190927

Done and introduce the new CLANG9 toolchain: https://github.com/lgao4/edk2/tree/CLANG9

We also enabled the "Cross OS/Host Reproducible Build": https://github.com/lgao4/edk2/tree/CLANG9#cross-oshost-reproducible-build

Enable clang COFF native build toolchain in edk2 - shijunjing/edk2 GitHub Wiki

20190704

20190705

20190711

20190715

20190803

20190805

20190806

20190809

20190816

20190817

20190826

20190902

20190903

20190927

⚠️ GitHub.com Fallback ⚠️

Enable clang COFF native build toolchain in edk2 - shijunjing/edk2 GitHub Wiki

20190704

20190705

20190711

20190715

20190803

20190805

20190806

20190809

20190816

20190817

20190826

20190902

20190903

20190927

⚠️ **GitHub.com Fallback** ⚠️

⚠️ GitHub.com Fallback ⚠️