Module level build cache enabling in edk2 - shijunjing/edk2 GitHub Wiki

This page focus on the design of module-level build cache, which is to reuse all module built results (including all intermediate artifacts, parsing result, makefile, obj files, debug symbol files, linking executable files, etc.) from the cache of a previous run, and not rebuild the module if possible. The work focus on below two questions:

1. How to define and track a edk2 module's content and its dependency content which grantee a deterministic build result.
2. How to quickly tell whether or not a module is cache miss or cache hit in a new clean build. In other words, to tell whether we need rebuild a module.

See the background details in https://github.com/shijunjing/edk2/wiki/Incremental-binary-build-enabling-in-edk2

• we can design a "Make Cache" solution which is to check cache before run the module Make and bases on the files list in the Makefile to calculate the hash. This Make cache design soundness is exactly same as current incremental build, because both of them base on same Makefile and have exactly same inputs information for every module. Details see the below 20190306 working notes.
• updated: we already added library cache support from Make cache into the binary cache and don't need a separated Make cache: https://github.com/shijunjing/edk2/wiki/Module-level-build-cache-enabling-in-edk2/#20190506

  D:\Project\Edk2Git\workspace> git clone https://github.com/shijunjing/edk2.git edk2-fork2
  D:\Project\Edk2Git\workspace> cd edk2-fork2
  D:\Project\Edk2Git\workspace\edk2-fork2>git checkout makecache
  D:\Project\Edk2Git\workspace\edk2-fork2>edksetup.bat ForceRebuild
  D:\Project\Edk2Git\workspace\edk2-fork2>edksetup.bat Reconfig
  D:\Project\Edk2Git\workspace\edk2-fork2>Build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86 GenMake

• https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Specifications
• clang-scan-deps: Fast dependency scanning for explicit modules: http://llvm.org/devmtg/2019-04/talks.html#Talk_16

• 20190301

LLVM linker owner happen to propose a linker based dependency graph feature, which is very helpful to capture the edk2 module's accurate dependency efficiently. See mail thread: http://lists.llvm.org/pipermail/llvm-dev/2019-February/130649.html

LLVM linker owner offer a POC patch on lld to output the module dependency: http://lists.llvm.org/pipermail/llvm-dev/2019-February/130661.html, and I'm testing the patch.

1. rebuild lld with above patch
2. $ export CLANG38_BIN=~/llvm/releasebuild/bin/
3. $ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -t CLANG38 -b NOOPT -m UefiCpuPkg/CpuMpPei/CpuMpPei.inf
4. the lld will print out a dot file to the stdout

But it looks include all function symbols and many of them is not real dependency. Need more investigation.

Current dependency result does not include the main symbol, and seems not correct. Sent to the owner to ask: http://lists.llvm.org/pipermail/llvm-dev/2019-March/130671.html

• 20190304

1. D:\Project\Edk2Git\workspace\edk2>edksetup.bat
2. D:\Project\Edk2Git\workspace\edk2>edksetup.bat Reconfig
3. D:\Project\Edk2Git\workspace\edk2>edksetup.bat Rebuild
4. D:\Project\Edk2Git\workspace\edk2>python D:\Project\Edk2Git\workspace\edk2\BaseTools\Source\Python\build\build.py -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86
5. D:\Project\Edk2Git\workspace\edk2>"C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.5\bin\pycharm64.exe"
6. New the project in D:\Project\Edk2Git\workspace\edk2\BaseTools and select the Python 2.7.14 in c:\Python27\python.exe as the project interpreter.
7. Add Debug and set Script path as "D:\Project\Edk2Git\workspace\edk2\BaseTools\Source\Python\build\build.py" and Parameters as "-p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86"

• 20190306

The BaseTools build/AutoGen parser finish parsing and break down monolithic platform meta files into module level meta files:

1. Makefile
2. AutoGenTimeStamp
3. OUTPUT\object_files.lst
4. OUTPUT\static_library_files.lst
5. DEBUG\AutoGen.c
6. DEBUG\AutoGen.h
7. DXE PCD database? (The DXEPcdDataBase.raw in edk2\Build\Ovmf3264\DEBUG_VS2015x86\X64\MdeModulePkg\Universal\PCD\Dxe\Pcd\OUTPUT will be moved to FV folder in Q2, so will no problem to not include it as a output of Pcd module Makefile run)

Then, begin to build the modules in multi-threads through run the Make at below function.

edk2\BaseTools\Source\Python\build\build.py line 473

                Bt.Start()

edk2\BaseTools\Source\Python\build\build.py line 637

## Start build task thread
#
def Start(self):
    EdkLogger.quiet("Building ... %s" % repr(self.BuildItem))
    Command = self.BuildItem.BuildCommand + [self.BuildItem.Target]
    self.BuildTread = Thread(target=self._CommandThread, args=(Command, self.BuildItem.WorkingDir))
    self.BuildTread.setName("build thread")
    self.BuildTread.setDaemon(False)
    self.BuildTread.start()

In the all above 7 meta files of a module, the Makefile has includes the AutoGenTimeStamp, OUTPUT\object_files.lst, and OUTPUT\static_library_files.lst as Makefile Build Macro. e.g. edk2\Build\Ovmf3264\DEBUG_VS2015x86\X64\MdeModulePkg\Core\Dxe\DxeMain\Makefile:

#
# Build Macro
#
OBJECT_FILES =  \
    $(OUTPUT_DIR)\SectionExtraction\CoreSectionExtraction.obj \
    $(OUTPUT_DIR)\Image\Image.obj \
    ...
    $(OUTPUT_DIR)\AutoGen.obj

STATIC_LIBRARY_FILES =  \
    $(BIN_DIR)\MdePkg\Library\DxePcdLib\DxePcdLib\OUTPUT\DxePcdLib.lib \
    $(BIN_DIR)\MdePkg\Library\BaseLib\BaseLib\OUTPUT\BaseLib.lib \
    ...
COMMON_DEPS = $(WORKSPACE)\MdePkg\Include\Library\UefiBootServicesTableLib.h \
              $(WORKSPACE)\MdePkg\Include\Protocol\Timer.h \
              $(WORKSPACE)\MdePkg\Include\Protocol\GuidedSectionExtraction.h \
              ...
              $(DEBUG_DIR)\AutoGen.h \
              ...

And the DEBUG\AutoGen.c and DEBUG\AutoGen.h are force included in the Makefile. So, the Makefile includes all possible input files and dependency for a module in fact! OK, then the we can design a build cache checkpoint before run the module Make and bases on the files list in the Makefile to calculate the hash. This build cache design soundness is exactly same as current incremental build, because both of them base on same Makefile and have exactly same inputs information.

It's better if BaseTools build/AutoGen parser can generate a stand alone file to include all above cache files together with the Makefile, then my module build cache tool can directly read the cache file list and not need to parse the makefile to get specific build Macro.

• 20190321

a. Create makecache branch and add the CacheOutputFile definition in BaseTools/Conf/build_rule.template

  D:\Project\Edk2Git\workspace\edk2-fork2>Build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86 GenMake
  build.py...
  d:\project\edk2git\workspace\edk2-fork2\conf\build_rule.txt(131): error 3000: Invalid syntax/format
          Unknown subsection: <CacheOutputFile>

Need update build.py to support the new rules.

b. Below LaunchCommand() is the single entry to invoke external program, including the make. We can add makecache checkpoint here to parse: 1. whether the command is "make". 2. whether the target is "tbuild".

  D:\Project\Edk2Git\workspace\edk2-fork2\BaseTools\Source\Python\build\build.py
  def LaunchCommand(Command, WorkingDir):
      ...
      Proc = None
      EndOfProcedure = None
      try:
          # launch the command
          Proc = Popen(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)
      ... 

Only support "tbuild" for now. And if user build single module with "build -m ...", which invoke "make mbuild" internally, the MakeCache will not work. Because MakeCache cannot catch complete input files for the mbuild build target. The mbuild build target depend on the gen_libs which will invoke make recursively in internal as below. Since the MakeCache only base on the current module files, which not recursively include the dependent libs files, to check the whether cache hit, it will miss the libs input files. Might enhance it in the future.

    mbuild: $(INIT_TARGET) $(BC_TARGET) gen_libs $(PCH_TARGET) $(CODA_TARGET)
    ... ...
    #
    # GenLibsTarget
    #
    gen_libs:
      @"$(MAKE)" $(MAKE_FLAGS) -f $(BIN_DIR)\MdePkg\Library\DxePcdLib\DxePcdLib\Makefile
      @"$(MAKE)" $(MAKE_FLAGS) -f $(BIN_DIR)\MdePkg\Library\BaseMemoryLibRepStr\BaseMemoryLibRepStr\Makefile
      @"$(MAKE)" $(MAKE_FLAGS) -f $(BIN_DIR)\MdePkg\Library\BaseLib\BaseLib\Makefile

c. Confirmed with build tool owner that current build --hash functionality only bypass the parser's output(e.g. AutoGen.c, makefile, etc.) and will still invoke the make to do the incremental build.

     D:\Project\Edk2Git\workspace\edk2-fork2>Build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86 --hash

• 20190325

Debug the build.py with PyCharm:

    D:\Project\Edk2Git\workspace\edk2-fork2>edksetup.bat
    D:\Project\Edk2Git\workspace\edk2-fork2>"C:\Program Files\JetBrains\PyCharm Community Edition 2018.2.5\bin\pycharm64.exe"

Create the Run/Debug Configurations:

set breakpoint in build.py line 220:

    Proc = Popen(Command, stdout=PIPE, stderr=PIPE, env=os.environ, cwd=WorkingDir, bufsize=-1, shell=True)

Run then pause at the Popen function:

• 20190326

Design the make cache and parser2fv cache together: https://soco.intel.com/docs/DOC-2599416

• 20190408

Work on deployment of the current edk2 build cache solution in CI. current edk2 build cache solution is the Parser2Module cache in below picture, which enabled by “Build --hash --binary-destination --binary-source” option.

1. The Parser2Fv cache can be useful for MinPlatform build since the MinPlatform has separated the platform monolithic FV into many small FVs. So, the FV level cache miss normally only happen in limited scope FVs.
2. The Parser2Module cache is useful for general platform and will cache hit if user’s patch not change platform level meta files, e.g. dsc, fdf.
3. The Make Cache is useful for general platform and will only begin to work after Parser2Module cache miss. The Make Cache will cache hit if user’s patch not change current build module related artifacts.

But current Parser2Module Cache has below limitations, and we will improve them in the future:

1. No complete output file definition, might miss PDB and other important output files.
2. No output files recovery filters and configures to fit different build scenarios.

Build with --hash --binary-destination option in windows:

  D:\Project\Edk2Git\workspace\edk2>Build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86 --hash --binary-destination=BINCACHEDEST
  ... ..
  FV Space Information
  SECFV [11%Full] 212992 total, 23520 used, 189472 free
  PEIFV [20%Full] 917504 total, 192616 used, 724888 free
  DXEFV [38%Full] 11534336 total, 4430728 used, 7103608 free
  FVMAIN_COMPACT [34%Full] 3440640 total, 1185696 used, 2254944 free

  - Done -
  Build end time: 10:27:54, Apr.08 2019
  Build total time: 00:12:34

The BINCACHEDEST is generated in D:\Project\Edk2Git\workspace\edk2\BINCACHEDEST

Remove the Build folder and clean build with --hash --binary-source:

  D:\Project\Edk2Git\workspace\edk2>rmdir Build /s /q
  D:\Project\Edk2Git\workspace\edk2>Build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t VS2015x86 --hash --binary-source=BINCACHEDEST
  ... ...
  FV Space Information
  SECFV [11%Full] 212992 total, 23520 used, 189472 free
  PEIFV [20%Full] 917504 total, 192616 used, 724888 free
  DXEFV [38%Full] 11534336 total, 4430296 used, 7104040 free
  FVMAIN_COMPACT [34%Full] 3440640 total, 1185608 used, 2255032 free

  - Done -
  Build end time: 10:33:44, Apr.08 2019
  Build total time: 00:01:25

Build with --hash --binary-destination option in Linux:

  jshi19@ub2-uefi-b01:~/wksp_efi/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t CLANG38 --hash --binary-destination=BINCACHEDEST
  ... ...
  FV Space Information
  SECFV [10%Full] 212992 total, 22048 used, 190944 free
  PEIFV [19%Full] 917504 total, 180136 used, 737368 free
  DXEFV [34%Full] 11534336 total, 3992064 used, 7542272 free
  FVMAIN_COMPACT [33%Full] 3440640 total, 1161792 used, 2278848 free

  - Done -
  Build end time: 10:42:12, Apr.08 2019
  Build total time: 00:01:46

Remove the Build folder and clean build with --hash --binary-source:

  jshi19@ub2-uefi-b01:~/wksp_efi/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t CLANG38 --hash --binary-source=BINCACHEDEST
  ... ...
  FV Space Information
  SECFV [10%Full] 212992 total, 22048 used, 190944 free
  PEIFV [19%Full] 917504 total, 180136 used, 737368 free
  DXEFV [34%Full] 11534336 total, 3991632 used, 7542704 free
  FVMAIN_COMPACT [33%Full] 3440640 total, 1161680 used, 2278960 free

  - Done -
  Build end time: 10:44:56, Apr.08 2019
  Build total time: 00:00:11

• Found a bug about --hash --binary-source=BINCACHEDEST that if the bin cache is built by CLANG38 toolchain, the GCC5 toolchain can directly use it as below. I believe this is not correct. --binary-source need check the toolchain info about current cache and cannot share different toolchain built cache for different toolchain. This issue is from the current BIN cache biggest limitation: not use hash value as cache index, but just simply the build output folder path.

  jshi19@ub2-uefi-b01:~/wksp_efi/edk2$ build -p OvmfPkg/OvmfPkgIa32X64.dsc -a IA32 -a X64 -b DEBUG -t GCC5 --hash --binary-source=BINCACHEDEST
  

Denlow platform normal build data:

  C:\TCWork\a47e097b213e846a>call Intel\DenlowPkg\b.bat
  ... ...
  - Done -
  Build end time: 20:25:56, Apr.07 2019
  Build total time: 00:11:40

Add --hash --binary-destination=c:\BINCACHEDESTin C:\TCWork\a47e097b213e846a\Intel\DenlowPkg\b.bat:

  call build -D PLATFORM_ARCH=%PLATFORM_ARCH% %MACRO_DEFINE% %PCD_SETTING% -t %TOOL_CHAIN_TAG% -a IA32 -a %PLATFORM_ARCH% -p DenlowPkg\DenlowPkg.dsc -b %TARGET% --log=%LOG_FILE%  -y %REPORT_FILE%  --hash --binary-destination=c:\BINCACHEDEST

Remove the Build folder and clean build again:

  C:\TCWork\a47e097b213e846a>call Intel\DenlowPkg\b.bat
  ... ...
  - Done -
  Build end time: 23:07:06, Apr.07 2019
  Build total time: 00:07:38

The BIN Cache is in C:\BINCACHEDEST

Add --hash --binary-source=BINCACHEDEST in C:\TCWork\a47e097b213e846a\Intel\DenlowPkg\b.bat:

  call build -D PLATFORM_ARCH=%PLATFORM_ARCH% %MACRO_DEFINE% %PCD_SETTING% -t %TOOL_CHAIN_TAG% -a IA32 -a %PLATFORM_ARCH% -p DenlowPkg\DenlowPkg.dsc -b %TARGET% --log=%LOG_FILE%  -y %REPORT_FILE%  --hash --binary-source=c:\BINCACHEDEST

Remove the Build folder and clean build again. The build fail when generate the MICROCODE_FV:

  ... ...
  Generating MICROCODE_FV FV
  #Return Value = 2
  GenFfs: ERROR 0001: Error opening file
    c:\tcwork\a47e097b213e846a\Build\DenlowPkgX64\DEBUG_VS2015x86\X64\Microcode.bin


  ### ['GenFfs', '-t', 'EFI_FV_FILETYPE_RAW', '-g', '197DB236-F856-4924-90F8-CDF12FB875F3', '-o', 'c:\\tcwork\\a47e097b213e846a\\Build\\DenlowPkgX64\\DEBUG_VS2015x86\\FV\\Ffs\\197DB236-F856-4924-90F8-CDF12FB875F3MICROCODE_FV\\197DB236-F856-4924-90F8-CDF12FB875F3.ffs', '-i', 'c:\\tcwork\\a47e097b213e846a\\Build\\DenlowPkgX64\\DEBUG_VS2015x86\\X64\\Microcode.bin']


  build.py...
   : error 7000: Failed to generate FFS



  build.py...
   : error 7000: Failed to execute command


  - Failed -
  Build end time: 23:21:10, Apr.07 2019
  Build total time: 00:04:34

Tried the patch for https://bugzilla.tianocore.org/show_bug.cgi?id=1680 on Denlow, but still same fail:

  Generating MICROCODE_FV FV
  #Return Value = 2
  GenFfs: ERROR 0001: Error opening file
    c:\tcwork\a47e097b213e846a\Build\DenlowPkgX64\DEBUG_VS2015x86\X64\Microcode.bin

• 20190410

Test current Parser2Module cache cannot work on WhitleyRp platform:

  C:\steven\edkrepo>edkrepo --help
  C:\steven\edkrepo>edkrepo manifest
  C:\steven\edkrepo>edkrepo clone WhitleyRp WhitleyRp
  C:\steven\edkrepo>cd WhitleyRp\Intel\
  C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -c ICX -b RELEASE -t VS15

Add "--hash --binary-destination=C:\steven\edkrepo\BINCACHEDEST" in ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py build_iafw() cmd string:

  diff --git a/ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py b/ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py

  @@ -131,6 +131,7 @@ class IAFWBuild(object):
           if self.sub_target == 'HYBRID':
               self.build_target = 'HYBRID'
           cmd = "build -D XMLCLI_COMPILE=%s -D MAX_SOCKET=%x -D MAX_CORE=%x -D MAX_THREAD=%x -D PCHTARGET=%s -D CPUTARGET=%s -D SKUCPU=%s -D SUB_TARGET=%s -D DEFINED_PLATFORM_FLAG=%s -D BUILD_TARGET=%s -D PLATFORM_PKG=%s -D SILICON_PKG=%s -D SECURITY_PKG=%s -D PLATFORM_RESTRICTED_PKG=%s -D PCH_PKG=%s -D RP_PKG=%s -D FSP_PKG=%s -t %s  -a IA32 -a X64 -p %s -D BUILD_WITH_LEGACY_STACK=TRUE -D GENERATE_NEWROM_PATH=FALSE %s "%(self.xmlcli_compile,self.max_socket,self.max_core,self.max_thread,self.pch_target,self.cpu_target,self.cpu_sku,self.sub_target,self.defined_platform_flag,self.build_target,self.platform_pkg,self.silicon_pkg,self.security_pkg,self.platform_restricted_pkg, self.pch_pkg,self.rp_pkg, self.fsp_pkg, self.tool_chain_tag,self.platform_pkg_dsc_path, self.report_log)
  +        cmd += '--hash --binary-destination=C:\steven\edkrepo\BINCACHEDEST

clean build:

    C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -c ICX -b RELEASE -t VS15

Then, change to "--hash --binary-source=C:\steven\edkrepo\BINCACHEDEST" in ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py build_iafw() cmd:

  diff --git a/ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py b/ServerPlatformPkg/BuildScript/Source/BuildStep/IAFWBuild.py

           cmd = "build -D XMLCLI_COMPILE=%s -D MAX_SOCKET=%x -D MAX_CORE=%x -D MAX_THREAD=%x -D PCHTARGET=%s -D CPUTARGET=%s -D SKUCPU=%s -D SUB_TARGET=%s -D DEFINED_PLATFORM_FLAG=%s -D BUILD_TARGET=%s -D PLATFORM_PKG=%s -D SILICON_PKG=%s -D SECURITY_PKG=%s -D PLATFORM_RESTRICTED_PKG=%s -D PCH_PKG=%s -D RP_PKG=%s -D FSP_PKG=%s -t %s  -a IA32 -a X64 -p %s -D BUILD_WITH_LEGACY_STACK=TRUE -D GENERATE_NEWROM_PATH=FALSE %s "%(self.xmlcli_compile,self.max_socket,self.max_core,self.max_thread,self.pch_target,self.cpu_target,self.cpu_sku,self.sub_target,self.defined_platform_flag,self.build_target,self.platform_pkg,self.silicon_pkg,self.security_pkg,self.platform_restricted_pkg, self.pch_pkg,self.rp_pkg, self.fsp_pkg, self.tool_chain_tag,self.platform_pkg_dsc_path, self.report_log)
  +        cmd += '--hash --binary-source=C:\steven\edkrepo\BINCACHEDEST

And clean build again:

    C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -cl
    C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -c ICX -b RELEASE -t VS15
    ... ...
    GenFds.py...
    : error 000E: File/directory not found in workspace
    Build\WhitleyRpPkg\RELEASE_VS2015x86\X64\Microcode.bin is not found in packages path:
    c:\steven\edkrepo\whitleyrp\intel\.
    c:\steven\edkrepo\whitleyrp\intel\.
    C:\steven\edkrepo\WhitleyRp\Edk2
    C:\steven\edkrepo\WhitleyRp\FDBin


    build.py...
    : error 7000: Failed to execute command
    GenFds -f c:\steven\edkrepo\whitleyrp\intel\WhitleyRpPkg\PlatformPkg.fdf --conf=c:\steven\edkrepo\whitleyrp\intel\Conf -o c:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\RELEASE_VS2015x86 -t VS2015x86 -b RELEASE -p c:\steven\edkrepo\whitleyrp\intel\WhitleyRpPkg\PlatformPkg.dsc -a IA32,X64  -D "EFI_SOURCE=c:\\steven\\edkrepo\\whitleyrp\\intel\\edkcompatibilitypkg"  -D "EDK_SOURCE=c:\\steven\\edkrepo\\whitleyrp\\intel\\edkcompatibilitypkg"  -D "PLATFORM_PKG=ServerPlatformPkg"  -D "MAX_THREAD=2"  -D "TOOLCHAIN=VS2015x86"  -D "ECP_SOURCE=c:\\steven\\edkrepo\\whitleyrp\\intel\\edkcompatibilitypkg"  -D "FAMILY=MSFT"  -D "FSP_PKG"  -D "MAX_CORE=64"  -D "PCHTARGET=LBG"  -D "DEFINED_PLATFORM_FLAG"  -D "WORKSPACE=c:\\steven\\edkrepo\\whitleyrp\\intel"  -D "SUB_TARGET"  -D "GENERATE_NEWROM_PATH=FALSE"  -D "PLATFORM_RESTRICTED_PKG=ServerRestrictedPkg"  -D "ARCH=IA32 X64"  -D "PCH_PKG=ServerSiliconPkg/Pch/SouthClusterLbg"  -D "CPUTARGET=ICX"  -D "TOOL_CHAIN_TAG=VS2015x86"  -D "TARGET=RELEASE"  -D "BUILD_TARGET=RELEASE"  -D "RP_PKG=WhitleyRpPkg"  -D "UNIFY_IFWI=FALSE"  -D "XMLCLI_COMPILE=TRUE"  -D "SECURITY_PKG=ServerSecurityPkg"  -D "EDK_TOOLS_PATH=c:\\steven\\edkrepo\\whitleyrp\\edk2\\basetools"  -D "BUILD_WITH_LEGACY_STACK=TRUE"  -D "SKUCPU"  -D "MAX_SOCKET=4"  -D "SILICON_PKG=ServerSiliconPkg" [C:\steven\edkrepo\WhitleyRp\Intel]

    - Failed -
    Build end time: 13:49:45, Apr.10 2019
    Build total time: 00:18:17

Tried use laptop machine to build Kabylake of MinPlatform: https://github.com/tianocore/edk2-platforms/tree/devel-MinPlatform. But it fails with nasm error:

          "C:\nasm\nasm" -Id:\project\edk2git\workspace\edk2\MdePkg\Library\BaseLib\Ia32\ -Ox -f win32 -g -DBDAT_SUPPORT=1 -DSG_SUPPORT=1 -DMEM_DOWN_FLAG=1 -DFSP_WRAPPER_FLAG -DDISABLE_NEW_DEPRECATED_INTERFACES=1 -DACPI_SUPPORT=1 -DMINTREE_FLAG=1 -o d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\IA32\MdePkg\Library\BaseLib\BaseLib\OUTPUT\Ia32\LongJump.obj d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\IA32\MdePkg\Library\BaseLib\BaseLib\OUTPUT\Ia32\LongJump.iii
  d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\IA32\MdePkg\Library\BaseLib\BaseLib\OUTPUT\Ia32\LongJump.iii:16: fatal: unable to open include file `Nasm.inc'
  NMAKE : fatal error U1077: 'C:\nasm\nasm.EXE' : return code '0x1'
  Stop.

But it builds pass in w10-uefi-b03 machine.

OK，the Kabylake just build pass on my laptop. The obove build failure is because I didn’t clean the old D:\Project\Edk2Git\workspace\Conf. Need remove the old Conf if it exist:

   D:\Project\Edk2Git\workspace>rmdir Conf /s /q
   D:\Project\Edk2Git\workspace>edk2\edksetup.bat
   D:\Project\Edk2Git\workspace>cd edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3
   D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
   D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep
   D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld

Next is to test the BIN Cache "--hash --binary-source --binary-source" whether works in MinPlatform Kabylake.

• 20190416

The BIN Cache "--hash --binary-source --binary-source" cannot works in MinPlatform Kabylake. Reproduce steps:

1. Follow the steps in below link to checkout Kabylake code in workspace: https://github.com/tianocore/edk2-platforms/tree/devel-MinPlatform

2. copy the bld.bat to bld1.bat and append the --hash --binary-destination as below:

    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>copy bld.bat bld1.bat
    --- a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld1.bat
    +++ b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld1.bat
    @@ -109,7 +109,7 @@ goto BUILD_FLAGS_LOOP
   
     @if %SILENT_MODE% EQU TRUE goto BldSilent
   
    -call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS%
    +call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS% --hash --binary-destination=C:\steven\BinCache
   
     @if %ERRORLEVEL% NEQ 0 goto BldFail
     @echo.
   

3. copy the bld.bat to bld2.bat and append the --hash --binary-destination in bld2.bat as below:

    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>copy bld.bat bld2.bat
    --- a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld2.bat
    +++ b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld2.bat
    @@ -109,7 +109,7 @@ goto BUILD_FLAGS_LOOP
   
     @if %SILENT_MODE% EQU TRUE goto BldSilent
   
    -call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS%
    +call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS% --hash --binary-source=C:\steven\BinCache
   
     @if %ERRORLEVEL% NEQ 0 goto BldFail
     @echo.
   

4. Clean build with bld1.bat firstly:

    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld1.bat
   

5. Clean build with bld2.bat again:

    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat
    C:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld2.bat
    ... ...
    Generating FVMICROCODE FV
    #Return Value = 2
    GenFfs: ERROR 0001: Error opening file
      c:\steven\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\MicrocodeUpdates.bin
   
    ### ['GenFfs', '-t', 'EFI_FV_FILETYPE_RAW', '-g', '197DB236-F856-4924-90F8-CDF12FB875F3', '-o', 'c:\\steven\\Build\\KabylakeOpenBoardPkg\\KabylakeRvp3\\DEBUG_VS2015x86\\FV\\Ffs\\197DB236-F856-4924-90F8-CDF12FB875F3FVMICROCODE\\197DB236-F856-4924-90F8-CDF12FB875F3.ffs', '-i', 'c:\\steven\\Build\\KabylakeOpenBoardPkg\\KabylakeRvp3\\DEBUG_VS2015x86\\X64\\MicrocodeUpdates.bin']
    build.py...
     : error 7000: Failed to generate FFS
    build.py...
     : error 7000: Failed to execute command
    - Failed -
   

This build failure direct reason is the --binary-destination only save the module level built outputs and not save the platform level output file: C:\steven\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\MicrocodeUpdates.bin. But the Kabylake fdf file edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3\OpenBoardPkg.fdf defines the FV.FvMicrocode to depende on the platform level output file X64/MicrocodeUpdates.bin as below. So when the --binary-source try to recovery the build output files from cache, the FV.FvMicrocode build fail because the X64/MicrocodeUpdates.bin is missing.

    FILE RAW = 197DB236-F856-4924-90F8-CDF12FB875F3 {
      $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/MicrocodeUpdates.bin
    }

If you continue to debug, you will see the the X64/MicrocodeUpdates.bin actually is copied from MicrocodeUpdates module build output folder $(OUTPUT_DIR):

    D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>call build -n 4   --hash --binary-destination=C:\steven\BINCACHEDEST -m D:\Project\Edk2Git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\MicrocodeUpdates.inf
    Build environment: Windows-10-10.0.16299-SP0
    Build start time: 14:38:57, Apr.16 2019

    WORKSPACE        = d:\project\edk2git\workspace
    PACKAGES_PATH    = d:\project\edk2git\workspace\edk2-platforms\platform\intel;d:\project\edk2git\workspace\edk2-platforms\silicon\intel;d:\project\edk2git\workspace\edk2-non-osi\silicon\intel;d:\project\edk2git\workspace\fsp;d:\project\edk2git\workspace\edk2;d:\project\edk2git\workspace
    EDK_TOOLS_PATH   = d:\project\edk2git\workspace\edk2\basetools
    CONF_PATH        = d:\project\edk2git\workspace\conf
    PYTHON_COMMAND   = py -3
    Architecture(s)  = IA32 X64
    Build target     = DEBUG
    Toolchain        = VS2015x86
    Active Platform          = d:\project\edk2git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3\OpenBoardPkg.dsc
    Active Module            = d:\project\edk2git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\MicrocodeUpdates.inf
    Flash Image Definition   = d:\project\edk2git\workspace\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3\OpenBoardPkg.fdf
    Processing meta-data ................ done!
    Generating code . done!
    Generating makefile . done!
    Building ... d:\project\edk2git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\MicrocodeUpdates.inf [X64]
            "GenFw" -o d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\KabylakeSiliconBinPkg\Microcode\MicrocodeUpdates\OUTPUT\MicrocodeUpdates.bin -j d:\project\edk2git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\mC0406E3_000000A0.mcb  d:\project\edk2git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\m80406E8_00000026.mcb  d:\project\edk2git\workspace\edk2-non-osi\Silicon\Intel\KabylakeSiliconBinPkg\Microcode\mC0806E9_00000030.mcb
            copy /y d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\KabylakeSiliconBinPkg\Microcode\MicrocodeUpdates\OUTPUT\MicrocodeUpdates.bin d:\project\edk2git\workspace\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\MicrocodeUpdates.bin
            1 file(s) copied.
    - Done -
    Build end time: 14:39:16, Apr.16 2019
    Build total time: 00:00:20

You might continue to ask why not change the Kabylake fdf file and let the FV.FvMicrocode depende on the OUTPUT\MicrocodeUpdates.bin file in the module output folder? This way can also remove the extra copy action in the build_rule.template:

    [Microcode-Binary-File]
        <InputFile>
            *.mcb
        <ExtraDependency>
            $(MAKE_FILE)
        <OutputFile>
            $(OUTPUT_DIR)(+)$(MODULE_NAME).bin
        <Command>
            "$(GENFW)" -o ${dst} -j $(MICROCODE_BINARY_FILES) $(GENFW_FLAGS)
            -$(CP) ${dst} $(BIN_DIR)(+)$(MODULE_NAME_GUID).bin

All is because of a fdf file limitation that fdf file doesn't support generally specify a path file to module level, like the $(OUTPUT_DIR) in build_rule.template. In this case, the $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/MicrocodeUpdates.bin in fdf has to been changed to like below:

  FILE RAW = 197DB236-F856-4924-90F8-CDF12FB875F3 {
   $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/KabylakeSiliconBinPkg/Microcode/MicrocodeUpdates/OUTPUT/MicrocodeUpdates.bin
  }

You can see above file path is very ugly and hardcode the 'OUTPUT' name of temp build folder in fdf file. So, we need support the Macros like $(OUTPUT_DIR),$(DEBUG_DIR),$(BUILD_DIR),$(BIN_DIR),$(FV_DIR) in fdf or even dsc files.

Have submitted the BZ for this issue: https://bugzilla.tianocore.org/show_bug.cgi?id=1723. And anyway, below patch can fix this issue:

    diff --git a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.fdf b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.fdf
    index dc2a5d7e12..86feef6773 100644
    --- a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.fdf
    +++ b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.fdf
    @@ -210,7 +210,7 @@ READ_LOCK_CAP      = TRUE
     READ_LOCK_STATUS   = TRUE

     FILE RAW = 197DB236-F856-4924-90F8-CDF12FB875F3 {
    -  $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/MicrocodeUpdates.bin
    +  $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/KabylakeSiliconBinPkg/Microcode/MicrocodeUpdates/OUTPUT/MicrocodeUpdates.bin
     }

     [FV.FvPreMemory]

Test the BIN Cache "--hash --binary-source --binary-source" on MinPlatform Kabylake and WhitleyRp platforms with latest edk2 again to verify below BIN Cache enhancement commits:

    db4d47fd3ae9cb4a80399b7ea4cfd55f1b22d180
    f2b5e04acad2bac7f11d97871cc68e8a0d312289
    1b8caf0d87eacc66aba70e8698c7bd1518bb7be1
    2914e8153dd31a6594862ac6c0c7e7e460737991

The MinPlatform Kabylake still fails with X64\MicrocodeUpdates.bin missing error. The root cause is same as https://bugzilla.tianocore.org/show_bug.cgi?id=1723

  Generating FVMICROCODE FV
  #Return Value = 2
  GenFfs: ERROR 0001: Error opening file
    c:\steven\Build\KabylakeOpenBoardPkg\KabylakeRvp3\DEBUG_VS2015x86\X64\MicrocodeUpdates.bin

  ### ['GenFfs', '-t', 'EFI_FV_FILETYPE_RAW', '-g', '197DB236-F856-4924-90F8-CDF12FB875F3', '-o', 'c:\\steven\\Build\\KabylakeOpenBoardPkg\\KabylakeRvp3\\DEBUG_VS2015x86\\FV\\Ffs\\197DB236-F856-4924-90F8-CDF12FB875F3FVMICROCODE\\197DB236-F856-4924-90F8-CDF12FB875F3.ffs', '-i', 'c:\\steven\\Build\\KabylakeOpenBoardPkg\\KabylakeRvp3\\DEBUG_VS2015x86\\X64\\MicrocodeUpdates.bin']

WhitleyRp can build pass with "--hash --binary-source --binary-source", but the pass looks just because the Bin Cache is always cache miss on WhitleyRp platform, so the Bin Cache doesn't really work in WhitleyRp. It looks there is bug or design issue to block the WhitleyRp increamental build really work, and most modules have to rebuild everytime.

So, let me summary current issues about Bin Cache:

1. Doesn't support cache the platform level built result, and might cause build fail with microcode module file missing in some platform. https://bugzilla.tianocore.org/show_bug.cgi?id=1723
2. Bin Cache only support to same single platform with single toolcahin with target built files, and cannot save the multipult platform and targets build files together. E.g. WhitleyRp\DEBUG_VS2015x86, KabylakeRvp3\RELEASE_VS2015x86. https://bugzilla.tianocore.org/show_bug.cgi?id=1726
3. It looks there is bug or design issue to block the WhitleyRp increamental build really work, and most modules have to rebuild everytime, which causes Bin Cache not useful in WhitleyRp.
4. if the Bin Cache is built by CLANG38 toolchain, the GCC5 toolchain can directly use it as below. I believe this is not correct. --binary-source need check the toolchain info about current cache and cannot share different toolchain built cache for different toolchain. https://bugzilla.tianocore.org/show_bug.cgi?id=1725

• Find a new WhitleyRp build failure. When continue to build DEBUG target after RELEASE target build without clean, the WhitleyRp DEBUG build fails with below error:

    C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -c ICX -b RELEASE -t VS15
    ... ...
    C:\steven\edkrepo\WhitleyRp\Intel>WhitleyRpPkg\PlatformBIOSBuild.py -c ICX -b DEBUG -t VS15
    ... ...
    04-18 08:20 DEBUG    ProcessorStartup.inc:246: error: unable to hash file ProcessorStartup.inc. Debug information may be unavailable.
    04-18 08:20 DEBUG    
    04-18 08:20 DEBUG    NMAKE : fatal error U1077: 'C:\NASM\nasm.EXE' : return code '0x1'
    04-18 08:20 DEBUG    Stop.
    04-18 08:20 DEBUG    "C:\NASM\nasm" -Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcessorStartup\Ia32\ -Ox -f win32 -g -DSV_HOOKS -DPCH_LBG_ASM -o c:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\DEBUG_VS2015x86\IA32\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG\OUTPUT\Ia32\ProcessorStartup.obj c:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\DEBUG_VS2015x86\IA32\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG\OUTPUT\Ia32\ProcessorStartup.iii
    04-18 08:20 DEBUG    PrintLib.c
    04-18 08:20 DEBUG    @c:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\DEBUG_VS2015x86\IA32\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG\OUTPUT\cc_resp.txt
    04-18 08:20 DEBUG    /nologo /arch:IA32 /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /GL /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm /Gw -D SUB_CH=1 -D IA32   -D DEBUG_CODE_BLOCK=1 -D PLATFORM_VARIABLE_ATTRIBUTES=0x7 -D SV_HOOKS=SV_HOOKS -D SV_HOOKS_CPU -D SV_OR_SDP_FLAG -D MEM_SV_HOOKS  -D WHITLEY_FLAG -D WHITLEY_ONLY -D XMLCLI_ENABLE -D IPMI_SUPPORT  -DTRAD_FLAG=1   -DPCH_SERVER_BIOS_FLAG=1 -D SC_PATH="Pch/SouthClusterLbg" -DSUS_WELL_RESTORE=1    -DSERVER_BIOS_FLAG=1     -D MAX_SOCKET=4 -D MAX_CORE=64 -D MAX_THREAD=2 -D MAX_IMC=4 -D MAX_MC_CH=2 -D MAX_SAD_RULES=24 -D MAX_DRAM_CLUSTERS=1 -D MAX_KTI_PORTS=3 -D MAX_IIO_STACK=6 -D MAX_LOGIC_IIO_STACK=8 -D LT_FLAG  -D SPARING_SCRATCHPAD_SUPPORT -D SCRATCHPAD_DEBUG   -DENHANCED_WARNING_LOG=1 -D EFI_PCI_IOV_SUPPORT -D WHEA_SUPPORT -D ICX_HOST -D A0_HOST -D B0_HOST -D MMCFG_BASE_ADDRESS=0x80000000 -D DISABLE_NEW_DEPRECATED_INTERFACES -D LRDIMM_SUPPORT -D DDRT_SUPPORT  -DIE_SUPPORT=1 -D IE_PATH="Ie/v1" -DASF_PEI=1 -DWS_ICC_SUPPORT=1 -DAMT_SUPPORT=1 -DME_WS_SUPPORT=1 -D ME_TESTMENU_FLAG=1 -DME_SPS_SUPPORT=1 -DME_SUPPORT_FLAG=1 -D ME_PATH="Me/MeLbg" -D ASPEED_ENABLE -D ESPI_ENABLE /wd4819 /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcessorStartup\Ia32 /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcessorStartup /Ic:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\DEBUG_VS2015x86\IA32\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG\DEBUG /IC:\steven\edkrepo\WhitleyRp\Edk2\MdePkg /IC:\steven\edkrepo\WhitleyRp\Edk2\MdePkg\Include /IC:\steven\edkrepo\WhitleyRp\Edk2\MdePkg\Include\Ia32 /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\SouthCluster /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcMemInit /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcMemInit\Chip10nm\Ddr5\Include /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcMemInit\Chip\Include /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\BaseMspChipLib /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Universal\Dxe\CrystalRidge /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcMemInit\Simulation\Include /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\SimRegisters /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\FpgaIpLib\Include /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Me /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Me\Ws /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Me\Policy /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Me\Policy\Ws /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Ie /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Include\Ie\Policy /Ic:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Pch\AcpiTables\Dsdt /IC:\steven\edkrepo\WhitleyRp\Edk2\UefiCpuPkg /IC:\steven\edkrepo\WhitleyRp\Edk2\UefiCpuPkg\Include /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Include /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMemoryCoreLib\Core /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMemoryCoreLib\Core\Include /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMemoryCoreLib\Chip /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMemoryCoreLib\Chip\Include /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMemoryCoreLib\Platform /Ic:\steven\edkrepo\whitleyrp\intel\CpRcPkg\Library\BaseMspCoreLib\Include /IC:\steven\edkrepo\WhitleyRp\Edk2\IntelFsp2Pkg /IC:\steven\edkrepo\WhitleyRp\Edk2\IntelFsp2Pkg\Include
    04-18 08:20 DEBUG    
    04-18 08:20 DEBUG    
    04-18 08:20 DEBUG    
    04-18 08:20 DEBUG    build.py...
    04-18 08:20 DEBUG    : error 7000: Failed to execute command
    04-18 08:20 DEBUG    C:\Program Files (x86)\Microsoft Visual Studio 14.0\Vc\bin\nmake.exe /nologo tbuild [c:\steven\edkrepo\whitleyrp\intel\Build\WhitleyRpPkg\DEBUG_VS2015x86\IA32\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG]
    04-18 08:23 DEBUG    
    04-18 08:23 DEBUG    
    04-18 08:23 DEBUG    build.py...
    04-18 08:23 DEBUG    : error F002: Failed to build module
    04-18 08:23 DEBUG    c:\steven\edkrepo\whitleyrp\intel\ServerSiliconPkg\Library\ProcessorStartup\ProcessorStartupLBG.inf [IA32, VS2015x86, DEBUG]
    04-18 08:23 DEBUG    
    04-18 08:23 DEBUG    - Failed -
    04-18 08:23 DEBUG    Build end time: 08:23:26, Apr.18 2019
    04-18 08:23 DEBUG    Build total time: 00:05:34
    04-18 08:23 DEBUG    
    04-18 08:23 DEBUG    [cmd=build -D XMLCLI_COMPILE=TRUE -D MAX_SOCKET=4 -D MAX_CORE=64 -D MAX_THREAD=2 -D PCHTARGET=LBG -D CPUTARGET=ICX -D SKUCPU= -D SUB_TARGET= -D DEFINED_PLATFORM_FLAG= -D BUILD_TARGET=DEBUG -D PLATFORM_PKG=ServerPlatformPkg -D SILICON_PKG=ServerSiliconPkg -D SECURITY_PKG=ServerSecurityPkg -D PLATFORM_RESTRICTED_PKG=ServerRestrictedPkg -D PCH_PKG=ServerSiliconPkg/Pch/SouthClusterLbg -D RP_PKG=WhitleyRpPkg -D FSP_PKG=  -t VS2015x86  -a IA32 -a X64 -p C:\steven\edkrepo\WhitleyRp\Intel\WhitleyRpPkg/PlatformPkg.dsc -D BUILD_WITH_LEGACY_STACK=TRUE -D GENERATE_NEWROM_PATH=FALSE  --hash --binary-destination=C:\steven\edkrepo\BinCache -n 5   -D UNIFY_IFWI=FALSE]
    04-18 08:23 DEBUG    	 - FAIL : 334219 ms
  

This build fail seems not related to the Bin Cache. I edkrepo sync --override the latest clean WhitleyRp, but the DEBUG build still fails.

OK. above build failure is related the nasm version and the latest version nasm can pass the build:

  c:\nasm>nasm.exe -v
  NASM version 2.14.02 compiled on Dec 26 2018

• Collect some Bin Cache performance data on KabylakeRvp3 of MinPlatform:

  diff --git a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/OpenBoardPkg.fdf 
  @@ -210,7 +210,7 @@ READ_LOCK_CAP      = TRUE
   READ_LOCK_STATUS   = TRUE
  
   FILE RAW = 197DB236-F856-4924-90F8-CDF12FB875F3 {
  -  $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/MicrocodeUpdates.bin
  +  $(OUTPUT_DIRECTORY)/$(TARGET)_$(TOOL_CHAIN_TAG)/X64/KabylakeSiliconBinPkg/Microcode/MicrocodeUpdates/OUTPUT/MicrocodeUpdates.bin^M
   }
  
   [FV.FvPreMemory]
  
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld1.bat
  Build total time: 00:02:59
  
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld2.bat
  Build total time: 00:00:34
  
  Add a useless blank in C:\steven\edk2\MdeModulePkg\Bus\Ata\AtaBusDxe\AtaBus.c
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld2.bat
  Build total time: 00:01:17
  
  Revert the AtaBus.c and add a useless blank in C:\steven\edk2-platforms\Platform\Intel\MinPlatformPkg\PlatformInit\PlatformInitDxe\PlatformInitDxe.c
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld2.bat
  Build total time: 00:01:17
  
  Add a useless blank in both above AtaBus.c and PlatformInitDxe.c
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>cln.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>GitEdk2MinKabylake.bat
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>prep.bat r
  c:\steven\edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3>bld2.bat
  Build total time: 00:01:17
  

You can see current Bin Cache doesn't cache the library modules build results, so any module change will trigger all the dependency libries rebuild again, which cost (00:01:17 - 00:00:34)/00:02:59 = ~25% build time! This is the monolithic issue in current Bin Cache design. Make Cache will help to address this issue.

• Next is try to deploy the Bin Cache in SSG Core CI firstly. The Bin Cache usage model in CI is like this:

1. The daily build server will build and update all platforms/targets build cache once a day and store them in a central build cache share folder. The single central share folder contain different platform in different subfolders, e.g. BinCache\WhitleyRp\DEBUG_VS2015x86, BinCache\KabylakeRvp3\RELEASE_VS2015x86
2. The build agents will retrieve the new version build cache (all share folder content) to replace the old build cache in idle time.
3. Both the central build cache share folder and build agents only keep one version build cache, and not backup the old version build cache to save disk size. New build cache always have more possiblity to cache hit than old one in such daily manner.
4. The daily build server and build agents should have exactly same build workspace path.
5. The build cache share folder is read only for build agents.
6. The build agents will build with “--binary-source” to consume the build cache in local by default.

• Bin Cache database should be read only for --binary-source usage

Find another issue which is important to ensure the Bin Cache can be trusted for CI build agents. I find the build with --binary-source will fail if set the Cache database as read only. I think Bin Cache database should be read only for --binary-source usage and make sure the build agent, who consume the cache, will not pollute the cache. I’ve submit a BZ for this issue: https://bugzilla.tianocore.org/show_bug.cgi?id=1732

Continue to debug the issue of https://bugzilla.tianocore.org/show_bug.cgi?id=1732 what is happening is that the file is being copied successfully and maintaining the read only attribute, and then a later, second copy attempt fails because the file already exists as read-only. If the cache file multiple copies is hard to avoid, I’m fine to not use read only as the cache file permission.

Raise new issue that Binary Cache should update the recovered file’s creation and modification times to the current time: https://bugzilla.tianocore.org/show_bug.cgi?id=1742

Verify the Binary cache can work well in PurleyOpenBoardPkg, and doesn't have the KabylakeOpenBoardPkg microcode issue in https://bugzilla.tianocore.org/show_bug.cgi?id=1723. Because the microcode file type of PurleyOpenBoardPkg is .inc but the KabylakeOpenBoardPkg is .mcb. So, add the suggestion in the BZ 1723 comment to enhance the BaseTool AutoGen to add .mcb as binary type which can fix the KabylakeOpenBoardPkg microcode issue.

PurleyOpenBoardPkg binary cache testing steps:

D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus>GitEdk2MinMtOlympus.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus>bld1.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus>rmdir D:\Project\Edk2Git\workspace\Build /s /q
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus>bld2.bat

ClevoOpenBoardPkg binary cache testing steps:

D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>GitEdk2Clevo.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>prep.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>bld1.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>cln.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>GitEdk2Clevo.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>prep.bat
D:\Project\Edk2Git\workspace\edk2-platforms\Platform\Intel\ClevoOpenBoardPkg\N1xxWU>bld2.bat

Invesitgate how to extend the binary cache to support library cache. The library cache missing issue is the major gap for current binary cache, and have 25% extra time cost when rebuild the all module dependency libraries if cache miss happen. See 20190418 working notes. The library cache is also the major difference between current Binary cache and Make Cache. So we can add library cache support from Make cache into the binary cache then don't need a seperated Make cache.

BaseTools\Source\Python\AutoGen\AutoGen.py

def CopyModuleToCache(self):
    ...
    if not self.IsLibrary:
        ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
        if os.path.exists(ModuleFile):
            shutil.copy2(ModuleFile, FileDir)
    if not self.OutputFile:
        Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget, self.ToolChain]
        self.OutputFile = Ma.Binaries
    if self.OutputFile:
        ...

Current CopyModuleToCache implmentation doesn't support to save the .lib as cache because the library module Ma.Binaries in above code is empty. Unlike the driver module, BaseTool doesn't define what're the output binaries for a libary module.

Add below patch to extend the binary cache to save library .lib file as cache

  diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
  index 31721a6f9f..e006a1c5ac 100644
  @@ -3906,6 +3908,12 @@ class ModuleAutoGen(AutoGen):
               ModuleFile = path.join(self.OutputDir, self.Name + '.inf')
               if os.path.exists(ModuleFile):
                   shutil.copy2(ModuleFile, FileDir)
  +        else:
  +            OutputDir = self.OutputDir.replace('\\', '/').strip('/')
  +            DebugDir = self.DebugDir.replace('\\', '/').strip('/')
  +            for Item in self.CodaTargetList:
  +                File = Item.Target.Path.replace('\\', '/').strip('/').replace(DebugDir, '').replace(OutputDir, '').strip('/')
  +                self.OutputFile.add(File)
           if not self.OutputFile:
               Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget, self.ToolChain]
               self.OutputFile = Ma.Binaries

For the Binary Cache deployment on EDK2 CI, some platforms need to update the platform build batch file (e.g. PurleyOpenBoardPkg and KabylakeOpenBoardPkg bld.bat). Below are the update reference:

edk2-platforms\Platform\Intel\KabylakeOpenBoardPkg\KabylakeRvp3\bld.bat

diff --git a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld.bat b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld.bat
index e0ad5eefdc..f9a12d565a 100644
--- a/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld.bat
+++ b/Platform/Intel/KabylakeOpenBoardPkg/KabylakeRvp3/bld.bat
@@ -68,6 +68,11 @@ copy /y /b %WORKSPACE_FSP_BIN%\KabylakeFspBinPkg\Fsp_Rebased_S.fd+%WORKSPACE_FSP
 @SET REBUILD_MODE=
 @SET BUILD_ROM_ONLY=

+@if not defined BINARY_CACHE_PATH (
+  echo Info: BINARY_CACHE_PATH is empty, use BinCache as default
+  SET BINARY_CACHE_PATH=BinCache
+)
+
 :: Loop through arguements until all are processed

 :BUILD_FLAGS_LOOP
@@ -93,6 +98,20 @@ copy /y /b %WORKSPACE_FSP_BIN%\KabylakeFspBinPkg\Fsp_Rebased_S.fd+%WORKSPACE_FSP
   shift
   goto BUILD_FLAGS_LOOP
 )
+
+@if "%~1" == "cache-produce" (
+  SET BINARY_CACHE_CMD_LINE= --hash --binary-destination=%BINARY_CACHE_PATH%
+  shift
+  goto BUILD_FLAGS_LOOP
+)
+
+@if "%~1" == "cache-consume" (
+  SET BINARY_CACHE_CMD_LINE= --hash --binary-source=%BINARY_CACHE_PATH%
+  shift
+  goto BUILD_FLAGS_LOOP
+)
+
+
 :: Unknown build flag.
 shift
 goto BUILD_FLAGS_LOOP
@@ -105,11 +124,12 @@ goto BUILD_FLAGS_LOOP
 @echo    SILENT_MODE = %SILENT_MODE%
 @echo    REBUILD_MODE = %REBUILD_MODE%
 @echo    BUILD_ROM_ONLY = %BUILD_ROM_ONLY%
+@echo    BINARY_CACHE_CMD_LINE = %BINARY_CACHE_CMD_LINE%
 @echo.

 @if %SILENT_MODE% EQU TRUE goto BldSilent

-call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS%
+call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS% %BINARY_CACHE_CMD_LINE%

 @if %ERRORLEVEL% NEQ 0 goto BldFail
 @echo.
@@ -128,7 +148,7 @@ call %WORKSPACE_PLATFORM%\%PROJECT%\postbuild.bat %BUILD_ROM_ONLY%
 @echo ************************************************************************ >> Build.log
 @echo. >> Build.log

-call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS% 1>>Build.log 2>&1
+call build -n %NUMBER_OF_PROCESSORS% %REBUILD_MODE% %EXT_BUILD_FLAGS% %BINARY_CACHE_CMD_LINE% 1>>Build.log 2>&1

 @if %ERRORLEVEL% NEQ 0 goto BldFail
 @echo. >> Build.log

edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus\bld.bat

diff --git a/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/bld.bat b/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/bld.bat
index a66d19e66e..eda749af36 100644
--- a/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/bld.bat
+++ b/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/bld.bat
@@ -16,6 +16,11 @@ REM Run setlocal to take a snapshot of the environment variables.  endlocal is c
 setlocal
 set SCRIPT_ERROR=0

+@if not defined BINARY_CACHE_PATH (
+  echo Info: BINARY_CACHE_PATH is empty, use BinCache as default
+  SET BINARY_CACHE_PATH=BinCache
+)
+
 REM ---- Do NOT use :: for comments Inside of code blocks() ----

 ::**********************************************************************
@@ -34,6 +39,15 @@ if /I "%1"=="clean" (
   goto :EOF
 )

+if /I "%1"=="cache-produce" (
+  set BINARY_CACHE_CMD_LINE= --hash --binary-destination=%BINARY_CACHE_PATH%
+)
+
+if /I "%1"=="cache-consume" (
+  set BINARY_CACHE_CMD_LINE= --hash --binary-source=%BINARY_CACHE_PATH%
+)
+
+
 shift
 GOTO :parseCmdLine

@@ -92,8 +106,8 @@ echo                          Build Start
 echo.
 echo --------------------------------------------------------------------
 echo.
-echo build %BUILD_CMD_LINE% --log=%BUILD_LOG% %BUILD_REPORT_FLAGS%
-call build %BUILD_CMD_LINE% --log=%BUILD_LOG% %BUILD_REPORT_FLAGS%
+echo build %BUILD_CMD_LINE% --log=%BUILD_LOG% %BUILD_REPORT_FLAGS% %BINARY_CACHE_CMD_LINE%
+call build %BUILD_CMD_LINE% --log=%BUILD_LOG% %BUILD_REPORT_FLAGS% %BINARY_CACHE_CMD_LINE%
 echo --------------------------------------------------------------------
 echo.
 echo                          Build End

edk2-platforms\Platform\Intel\PurleyOpenBoardPkg\BoardMtOlympus\prebuild.bat

diff --git a/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/prebuild.bat b/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/prebuild.bat
index 81a9634d51..880e6417ac 100644
--- a/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/prebuild.bat
+++ b/Platform/Intel/PurleyOpenBoardPkg/BoardMtOlympus/prebuild.bat
@@ -188,8 +188,8 @@ set PRE_BUILD_CMD_LINE=%BUILD_CMD_LINE% -D MAX_SOCKET=%MAX_SOCKET%
 set PRE_BUILD_LOG=%WORKSPACE%\Build\prebuild.log
 set PRE_BUILD_REPORT=%WORKSPACE%\Build\preBuildReport.txt

-echo build %PRE_BUILD_CMD_LINE% -m %BOARD_PKG%\Acpi\BoardAcpiDxe\Dsdt.inf -y %PRE_BUILD_REPORT% --log=%PRE_BUILD_LOG%
-call build %PRE_BUILD_CMD_LINE% -m %BOARD_PKG%\Acpi\BoardAcpiDxe\Dsdt.inf -y %PRE_BUILD_REPORT% --log=%PRE_BUILD_LOG%
+echo build %PRE_BUILD_CMD_LINE% -m %BOARD_PKG%\Acpi\BoardAcpiDxe\Dsdt.inf -y %PRE_BUILD_REPORT% --log=%PRE_BUILD_LOG% %BINARY_CACHE_CMD_LINE%
+call build %PRE_BUILD_CMD_LINE% -m %BOARD_PKG%\Acpi\BoardAcpiDxe\Dsdt.inf -y %PRE_BUILD_REPORT% --log=%PRE_BUILD_LOG% %BINARY_CACHE_CMD_LINE%
 if %ERRORLEVEL% NEQ 0 EXIT /b %ERRORLEVEL%

 @REM PSYS == FIX0
(END)

For KabylakeOpenBoardPkg and PurleyOpenBoardPkg bld.bat, I’ve submitted BZs as below to update them to support Binary Cache. The reference patch already in the BZ comments.

Extend options in the Kabylake build bld.bat file to support Binary Cache https://bugzilla.tianocore.org/show_bug.cgi?id=1784

Extend options in the Purley build bld.bat file to support Binary Cache https://bugzilla.tianocore.org/show_bug.cgi?id=1785

How to send patch serial to edk2 devel mail list: Open the git bash, then git send-email *.patch --to … --cc

jshi19@jshi19-MOBL MINGW64 /d/Project/Edk2Git/workspace/edk2-platforms/New folder (devel-MinPlatform-checkin)
$ git send-email *.patch --to [email protected] --cc [email protected] --cc [email protected] --cc [email protected] --cc [email protected] --cc [email protected]

Continue working on extending the binary cache to support library cache：https://bugzilla.tianocore.org/show_bug.cgi?id=1797

Current problem is the cached .lib file is recovered before the library module AutoGen.h/AutoGen.c and makefile creation. So, even though fix the BZ 1742 issue (https://bugzilla.tianocore.org/show_bug.cgi?id=1742), the makefile time stamp is still always newer than the .lib file, which cause make will rebuild all the library files. In below code, the .lib is recovered in Ma.CanSkipbyHash(), but later on the library module makefile is created in Ma.CreateMakeFile(True), which cause the makefile timestamp is always newer than .lib file.

BaseTools\Source\Python\build\build.py:

def _BuildModule(self):
  ... ...
  for Module in Pa.Platform.Modules:
    ...
    Ma = ModuleAutoGen(Wa, Module, BuildTarget, ToolChain, Arch, self.PlatformFile)
    ...
    if Ma.CanSkipbyHash(): # recover the .lib file
        self.HashSkipModules.append(Ma)
        continue
            ...
            Ma.CreateCodeFile(True) # create the AutoGen.h/AutoGen.c
            ...
            Ma.CreateMakeFile(True) # create the makefile

Work on the below BZ 1806, and figure out a solution, see the BZ comment.

Different platform caches might overlay each other inside Binary Cache database if they use same platform name https://bugzilla.tianocore.org/show_bug.cgi?id=1806

• Figure out a library cache solution and attach the patch in below BZ:

Extend the binary cache to support library cache https://bugzilla.tianocore.org/show_bug.cgi?id=1797

• If the pip is not in the path env variable, use the complete pip path in python scripts folder:

  C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools>pip
  'pip' is not recognized as an internal or external command,
  operable program or batch file.
  
  C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools>C:\Python27\Scripts\pip install requests
  

• If python script report cannot import subfolder module, set the pythonpath evn variable:

  C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools>c:\Python27\python.exe TCscripts\server\build\uploadBinaryCache.py ...
  Traceback (most recent call last):
    File "TCscripts\server\build\uploadBinaryCache.py", line 12, in <module>
      from TCscripts.server.utils import iartifactory as arti
  ImportError: No module named TCscripts.server.utils
  
  C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools>set pythonpath=C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools
  
  C:\Users\sys_tianobui\Desktop\Test\iafw-cr-tools>c:\Python27\python.exe TCscripts\server\build\uploadBinaryCache.py ...
  Cache Folder Path: C:\TCWork\5b265536bf756d85\Edk2\BinCache
  Zip File Name: 2019-05-16_072646_BinCache
  

Deploy the build cache in edk2 CI: https://soco.intel.com/blogs/jshi19sBlog/2019/05/21/enable-build-cache-in-edk2-ci

The cache upload and download can succeed with same cache zip file, see 20190522 working note: https://soco.intel.com/blogs/jshi19sBlog/2019/05/21/enable-build-cache-in-edk2-ci

Sent patches for BZ 1797, 1742 and 1788:

[edk2-devel] [PATCH] BaseTools:Extend the binary cache to support library cache: https://edk2.groups.io/g/devel/topic/patch_basetools_extend_the/31819658?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31819658

[edk2-devel] [PATCH] BaseTools:Update binary cache restore time to current time: https://edk2.groups.io/g/devel/topic/patch_basetools_update/31819590?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31819590

[edk2-devel] [PATCH] BaseTools:Fix the library dependency missing in Binary Cache: https://edk2.groups.io/g/devel/topic/patch_basetools_fix_the/31819545?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31819545

Minimum Platform new URL in edk2: https://github.com/tianocore/edk2-platforms/tree/master/Platform/Intel

[PATCH v3 1/1] BaseTools:Extend the binary cache to support library cache: https://edk2.groups.io/g/devel/topic/patch_v3_1_1/31843505?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,31843505

Figure out patch for below two issues:

1. Build cache cannot store the cache files for library package: https://bugzilla.tianocore.org/show_bug.cgi?id=1867
2. Build Cache need output notification message if module cache miss & cache hit happen: https://bugzilla.tianocore.org/show_bug.cgi?id=1868

diff --git a/BaseTools/Source/Python/AutoGen/AutoGen.py b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -26,6 +26,7 @@ from io import BytesIO
 from .StrGather import *
 from .BuildEngine import BuildRule
 import shutil
+import filecmp
 from Common.LongFilePathSupport import CopyLongFilePath
 from Common.BuildToolError import *
 from Common.DataType import *
@@ -3571,6 +3572,10 @@ class ModuleAutoGen(AutoGen):
 
         # Skip the following code for libraries
         if self.IsLibrary:
+            try:
+                self.CopyModuleToCache()
+            except:
+                pass
             return
 
         # Skip the following code for modules with no source files
@@ -3916,6 +3921,7 @@ class ModuleAutoGen(AutoGen):
             Ma = self.BuildDatabase[self.MetaFile, self.Arch, self.BuildTarget, self.ToolChain]
             self.OutputFile = Ma.Binaries
         if self.OutputFile:
+            EdkLogger.quiet("Cache store ... %s[%s]" % (self.MetaFile.Path,self.Arch))
             for File in self.OutputFile:
                 File = str(File)
                 if not os.path.isabs(File):
@@ -3926,6 +3932,7 @@ class ModuleAutoGen(AutoGen):
                     destination_dir = os.path.dirname(destination_file)
                     CreateDirectory(destination_dir)
                     shutil.copy2(File, destination_dir)
+                    EdkLogger.quiet("Store file ... %s\\%s" % (destination_dir,File))
 
     def AttemptModuleCacheCopy(self):
         # If library or Module is binary do not skip by hash
@@ -3947,17 +3954,23 @@ class ModuleAutoGen(AutoGen):
                     for root, dir, files in os.walk(FileDir):
                         for f in files:
                             if self.Name + '.hash' in f:
-                                shutil.copy(HashFile, self.BuildDir)
+                                dst = path.join(self.BuildDir, self.Name + '.hash')
+                                if not os.path.exists(dst) or not filecmp.cmp(HashFile, dst):
+                                    shutil.copy(HashFile, self.BuildDir)
+                                    EdkLogger.quiet("Cache hit&restore ... %s[%s]" % (self.MetaFile.Path,self.Arch))
                             else:
                                 File = path.join(root, f)
                                 sub_dir = os.path.relpath(File, FileDir)
                                 destination_file = os.path.join(self.OutputDir, sub_dir)
                                 destination_dir = os.path.dirname(destination_file)
-                                CreateDirectory(destination_dir)
-                                shutil.copy(File, destination_dir)
+                                if not os.path.exists(destination_file) or not filecmp.cmp(File, destination_file):
+                                    CreateDirectory(destination_dir)
+                                    shutil.copy(File, destination_dir)
+                                    EdkLogger.quiet("Cache hit&restore ... %s[%s]" % (self.MetaFile.Path,self.Arch))
                     if self.Name == "PcdPeim" or self.Name == "PcdDxe":
                         CreatePcdDatabaseCode(self, TemplateString(), TemplateString())
                     return True
+        EdkLogger.quiet("Cache miss ... %s[%s]" % (self.MetaFile.Path, self.Arch))
         return False
 
     ## Create makefile for the module and its dependent libraries

Found new issue that the library build cache is not complete for both arches. When build with multipul arch, e.g. IA32 and X64, the same name libray cache can only be gnerated in one arch. Below is the reproduce steps:

Build the OvmfPkgIa32X64 platform with build cache enabled:

D:\Project\Edk2Git\workspace\edk2>build -p OvmfPkg\OvmfPkgIa32X64.dsc -a IA32 -a X64 -t VS2015x86 --hash --binary-destination=BinCache -y buildreport_ia32x64.txt

Check the BinCache folder that only BinCache\Build\OvmfIa32\DEBUG_VS2015x86\IA32\MdePkg\Library\BaseLib exist and the BinCache\Build\OvmfIa32\DEBUG_VS2015x86\X64\MdePkg\Library\BaseLib is missing.

Find above issue is because the set() can only contain single same class path ModuleAutoGen object, the IA32 and X64 ModuleAutoGen objects have same class path and the set() will see them as duplicated elements. So, the earlier added MA object will block the later added same calss path MA object. Can simply solve this issue with below patch:

diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py
@@ -2271,7 +2271,6 @@ class Build():
             RemoveDirectory(os.path.dirname(GlobalData.gDatabasePath), True)

     def CreateAsBuiltInf(self):
-        all_lib_set = set()
         all_mod_set = set()
         for Module in self.BuildModules:
             Module.CreateAsBuiltInf()
@@ -2282,10 +2281,7 @@ class Build():
         # Create a list of unique libs to copy
         for Module in all_mod_set:
             for lib in Module.LibraryAutoGenList:
-                all_lib_set.add(lib)
-        for lib in all_lib_set:
-            lib.CreateAsBuiltInf(True)
-        all_lib_set.clear()
+                lib.CreateAsBuiltInf(True)^M
         all_mod_set.clear()
         self.BuildModules = []
         self.HashSkipModules = []

• Build cache cannot store the cache files for library package：https://bugzilla.tianocore.org/show_bug.cgi?id=1867. FixPatch: https://edk2.groups.io/g/devel/topic/patch_v2_1_1/32013383?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,32013383

• Basetool need CopyFileOnChange function to avoid cache file writing race in multi-thread build: https://bugzilla.tianocore.org/show_bug.cgi?id=1894

• Build cache cannot store cache for the same library modules in different arch together: https://bugzilla.tianocore.org/show_bug.cgi?id=1895. FixPatch： https://edk2.groups.io/g/devel/topic/patch_basetools_cannot/32013233?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,32013233

• Change the build cache option prefix name from --binary- to --cache-: https://bugzilla.tianocore.org/show_bug.cgi?id=1896

• Be working on BZ1894: Basetool need CopyFileOnChange function to avoid cache file writing race in multi-thread build (https://bugzilla.tianocore.org/show_bug.cgi?id=1894)

https://github.com/meteotest/zarr/commit/a5dc7b30350ac04e8b26a928c0d5833bb819de70
https://github.com/zarr-developers/zarr/issues/263
avoid race condition during chunk write
When the chunk file is first removed before the new version
is moved into place, racing reads may encounter a missing chunk.
Using rename() or replace() without remove() avoids the issue
on Posix-Systems as the methods are atomic. The fallback of
remove() -> rename() is included for Windows pre Python 3.3.

I’ve sent below four BZs patches, I need check-in them quickly, please help to review them：

BZ: Build cache cannot store the cache files for library package(https://bugzilla.tianocore.org/show_bug.cgi?id=1867)
Patch: https://edk2.groups.io/g/devel/message/42176 

BZ: Build Cache need output notification message if module cache miss & cache hit happen(https://bugzilla.tianocore.org/show_bug.cgi?id=1868)
Patch: https://edk2.groups.io/g/devel/message/42435

BZ: Basetool need CopyFileOnChange function to avoid cache file writing race in multi-thread build(https://bugzilla.tianocore.org/show_bug.cgi?id=1894)
Patch: https://edk2.groups.io/g/devel/message/42436 
 
BZ: Build cache cannot store cache for the same library modules in different arch together(https://bugzilla.tianocore.org/show_bug.cgi?id=1895)
Patch: https://edk2.groups.io/g/devel/message/42438 

Follow the PurleyOpenBoardPkg BoardMtOlympus python building steps in https://github.com/tianocore/edk2-platforms/tree/master/Platform/Intel, and found the toolchain default is VS2015, not VS2015x86, which cause build fails with MSVC lib.exe path cannot found error. Need update the build_bios.py as below:

diff --git a/Platform/Intel/build_bios.py b/Platform/Intel/build_bios.py
@@ -198,7 +198,7 @@ def pre_build(build_config, build_type="DEBUG", silent=False, toolchain=None):
         config["TOOL_CHAIN_TAG"] = toolchain
     elif config.get("TOOL_CHAIN_TAG") is None:
         if os.name == 'nt':
-            config["TOOL_CHAIN_TAG"] = "VS2015"
+            config["TOOL_CHAIN_TAG"] = "VS2015x86"
         else:
             config["TOOL_CHAIN_TAG"] = "GCC5"

or

C:\steven\edk2-platforms\Platform\Intel>python build_bios.py -p BoardMtOlympus -t VS2015x86

Test the build cache on PurleyOpenBoardPkg BoardMtOlympus as below, and it works!

C:\steven\edk2-platforms\Platform\Intel>python build_bios.py --cleanall
C:\steven\edk2-platforms\Platform\Intel>python build_bios.py -t VS2015x86 -p KabylakeRvp3 --hash --binary-destination=BinCache
C:\steven\edk2-platforms\Platform\Intel>python build_bios.py -t VS2015x86 -p BoardMtOlympus --hash --binary-destination=BinCache
C:\steven\edk2-platforms\Platform\Intel>python build_bios.py --cleanall
C:\steven\edk2-platforms\Platform\Intel>python build_bios.py -t VS2015x86 -p KabylakeRvp3 --hash --binary-source=BinCache
C:\steven\edk2-platforms\Platform\Intel>python build_bios.py -t VS2015x86 -p BoardMtOlympus --hash --binary-source=BinCache

Just sent two new version patch for BZ 1984 and BZ 1895. Please help to review.

1. BZ: Basetool need CopyFileOnChange function to avoid cache file writing race in multi-thread build(https://bugzilla.tianocore.org/show_bug.cgi?id=1894) V2 Patch: https://edk2.groups.io/g/devel/message/42496

2. BZ: Build cache cannot store cache for the same library modules in different arch together(https://bugzilla.tianocore.org/show_bug.cgi?id=1895) V3 Patch: https://edk2.groups.io/g/devel/message/42494

1. Change the build cache option prefix name from --binary- to --cache- (https://bugzilla.tianocore.org/show_bug.cgi?id=1896). Patch: https://edk2.groups.io/g/devel/message/42521

report below two new BZs:

1. Build cache cannot store .efi and .pdb files in second incremental build https://bugzilla.tianocore.org/show_bug.cgi?id=1915

2. GenFds multi-thread build fails if enable the build cache https://bugzilla.tianocore.org/show_bug.cgi?id=1923

report below three new BZs:

1. Build cache need more comprehensive info to explain why cache miss https://bugzilla.tianocore.org/show_bug.cgi?id=1925

2. Build Cache very easily miss because of monolithic platform hash https://bugzilla.tianocore.org/show_bug.cgi?id=1927

3. Windows debug build is not reproducible even in the same path workspace https://bugzilla.tianocore.org/show_bug.cgi?id=1933

Bob's wiki, very useful for the build cache missing (BZ 1927) solution: https://github.com/BobCF/edk2/wiki/Multiple-thread-AutoGen

Be working on the BZ 1925 to print more info for cache miss. Meet a unstable file copy exception that CopyFileOnChange sometimes will fail with message: "WindowsError: [Error 32] The process cannot access the file because it is being used by another process". Below is the exception call stack:

  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\build\build.py", line 2119, in _MultiThreadBuildPlatform
  Ma.CreateMakeFile(True)
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\Common\caching.py", line 24, in CallMeHere
  Value = self._function(obj, *args,**kwargs)
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line 3996, in CreateMakeFile
  if Makefile.Generate():
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\GenMake.py", line 192, in Generate
  FileContent = self._TEMPLATE_.Replace(self._TemplateDict)
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\GenMake.py", line 473, in _TemplateDict
  self.ProcessDependentLibrary()
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\GenMake.py", line 1088, in ProcessDependentLibrary
  if not LibraryAutoGen.IsBinaryModule and not LibraryAutoGen.CanSkipbyHash():
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line 4174, in CanSkipbyHash
  if GlobalData.gBinCacheSource and self.AttemptModuleCacheCopy():
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\AutoGen\AutoGen.py", line 3947, in AttemptModuleCacheCopy
  CopyFileOnChange(File, destination_dir)
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\Common\Misc.py", line 548, in CopyFileOnChange
  os.rename(tempname, DstFile)
  File "C:\steven\edkrepo\WhitleyRp\edk2\BaseTools\Source\Python\Common\LongFilePathOs.py", line 49, in rename
  return os.rename(LongFilePath(old), LongFilePath(new))
  WindowsError: [Error 32] The process cannot access the file because it is being used by another process 

After some investigation, I found it is related to the file access through windows share folder in network (e.g. \10.239.219.158\jshi19\wksp_efi) . This issue is very easy to happen when I use beyondcompare to sync the remote code between the local WhitleyRp and linux VM. If I keep the beyondcompare opened to compare some Basetool python files in share folder (\10.239.219.158\jshi19\wksp_efi\edk2-2\BaseTools) and build the WhitleyRp at the same time, the WindowsError: [Error 32] issue will rais easily. If I close the beyondcompare and build the WhitleyRp, then the build will pass.

• so, a suggestion to build WhiteyRp is don't access the share folder in network when WhiteyRp is building!

Need to cherry-pick two below patches from edk2 to support "INF RuleOverride = MICROCODE $(PLATFORM_SI_BIN_PACKAGE)/Microcode/MicrocodeUpdates.inf"

• 04797875d1d53e53e31eca4fa01bc3cf6929f39c
• 6b74ccf0f8b53256b696ce921d50b873f6f9db57

Current binary cache bases on the '--hash' featrue which depends on the monolithic platform and package meta files （dsc, fdf, dec), and cause the cache is very easy to miss. I collect some build cache data about current module dependency files number and percentage on WhitleyRp and Ovmf platform. Overall, Ovmf module depends on 30~35% of all the platform build files, and WhitleyRp module depends on **~70% ** of all the platform build files which is very bad. We should find a better way to more accurately collect the WhitleyRp module dependency files.

• Original

1. WhitleyRp: all module number and dependency files number: "modules number: 1215", "files number: 18488", PciBus module dependency files number: ~13000, 13000/18488= 70.31% DxeCore module dependency files number: ~13000, 13000/18488= 70.31%
2. OvmfPkgIa32X64: all module number and dependency files number: "modules number:265", "files number:3049", PciBus module dependency files number and percentage: ~925, 925/3049= 30.33% DxeCore module dependency files number and percentage: ~1075, 1075/3049= 35.25%

• Add header files filter as below to only keep ('.h', '.inc', '.in') as pakcage interface files: for Root, Dirs, Files in os.walk(str(inc)): for File in sorted(Files, key=lambda x: str(x)): if not File.lower().endswith(('.h', '.inc', '.in')): continue

1. WhitleyRp: all module number and dependency files number: "modules number: 1215", "files number: 17140", PciBus module dependency files number: ~11500, 11500/17140= 67.9% DxeCore module dependency files number: ~11500, 11500/17140= 67.9%

2. OvmfPkgIa32X64: all module number and dependency files number: "modules number:265", "files number: 3048", PciBus module dependency files number and percentage: ~925, 925/3048= 30.34% DxeCore module dependency files number and percentage: ~1075, 1075/3048= 35.26%

So, to solve the high rate cahce missing issue, rewrite the module hash algorithms to implement the Make Cache: https://github.com/shijunjing/edk2/tree/buildcache. This make cache only depends on the module Makefile information which much more fine-grain.

def GenModuleHash(self):
...
        # Add Makefile
        if self.Makefile is not None:
            MakefileType = self.Makefile._FileType
            MakefileName = self.Makefile._FILE_NAME_[MakefileType]
            MakefilePath = os.path.join(self.MakeFileDir, MakefileName)
            DependencyFileSet.add(MakefilePath)
            # Add header files
            if self.Makefile.DependencyHeaderFileSet:
                for File in self.Makefile.DependencyHeaderFileSet:
                    DependencyFileSet.add(File)
        # Add AutoGen files
        if self.AutoGenFileList:
            for File in set(self.AutoGenFileList):
                DependencyFileSet.add(File)
        # Add Module Meta file
        DependencyFileSet.add(self.MetaFile)
        # Add Module's source files
        if self.SourceFileList:
            for File in set(self.SourceFileList):
                DependencyFileSet.add(File)
        # Add all Library dependency files
        if self.LibraryAutoGenList:
            for Lib in self.LibraryAutoGenList:
                if Lib not in GlobalData.gModuleHash:
                    Lib.GenModuleHash()
                for (File, Hash) in GlobalData.gModuleHashChain[Lib]:
                    DependencyFileSet.add(File)
                #m.update(GlobalData.gModuleHash[Lib].encode('utf-8'))
...

Make cache greatly improve the cache hit rate. But it depends on the makefile and have to wait for the whole autogen GenC/GenMake finish. In my test, it causes the build cache become slow when the most build modules can cache hit. And the make cache module hash is generated and checked one by one now which bring significant overhead into the autogen (4+ min in whitley). So, we need to enhance the autogen from single process to multipul process. Bob already offer a multipul process autogen solution in the branch: https://github.com/BobCF/edk2/tree/MAuto2. I'm working on porting the make cache to this branch.

• Have submitted a feature request BZ: Build cache need to work in multiple process way https://bugzilla.tianocore.org/show_bug.cgi?id=1975

Add GenLibHashChainInChildProcess to build all lib in parallel. Now build all modules as library in child process succussfully. But still need a build process queue to do the module final hash value and check in parallel way.

\\10.239.219.158\jshi19\wksp_efi\bfeng1\edk2\BaseTools\Source\Python\AutoGen\AutoGenWorker.py
    def test_run(self):
            ... ...
            Ma = ModuleAutoGen(self.Wa,module_metafile,target,toolchain,arch,PlatformMetaFile,self.data_pipe)
            Ma.IsLibrary = IsLib
            Ma.CreateCodeFile()
            Ma.CreateMakeFile(GenFfsList=FfsCmd.get((Ma.MetaFile.File, Ma.Arch),[]))
            #print(str(Ma))
            #EdkLogger.quiet("EdkLogger.quiet: %s" % str(Ma))
            Ma.GenLibHashChainInChildProcess(self.share_data)

I've pushed the code to my edk2 repository branch MAuto2_buildcache as below.

jshi19@ub2-uefi-b01:~/wksp_efi/bfeng1/edk2$ git push steven MAuto2_buildcache
jshi19@ub2-uefi-b01:~/wksp_efi/bfeng1/edk2$ git remote show steven
* remote steven
  Fetch URL: https://github.com/shijunjing/edk2.git
  Push  URL: https://github.com/shijunjing/edk2.git
  HEAD branch: master
  Remote branches:
    GCC61_v1             new (next fetch will store in remotes/steven)
    MAuto2               tracked
    MAuto2_buildcache    tracked
  ...

The build cache has a problem that when the cache source folder path is too long, the windows cannot get the cache file. This problem can be solved by enable the win32 long paths option as below: https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing