Allow map tiles to appear above sprites (so NPCs can walk behind tiles) with PRIORITY colors - pret/pokecrystal GitHub Wiki

Usually on overworld maps, sprites go on top of tiles. But there are exceptions: grass tiles overlap you as you walk, and the popup signs with location names (made of tiles) appear above all the NPC sprites.

Is it possible to make any tile in a tileset appear above sprites? Yes.

(The code for this feature was adapted from Pokémon Polished Crystal.)

Contents

  1. Some background information
  2. Define PAL_BG_PRIORITY_* constants
  3. Use one byte per color for tileset palettes
  4. Fix the skipped space in palette_map.asm files
  5. Fix the bank overflow
  6. Correctly read the extended palette data

1. Some background information

We can extend the tileset palette system to enable tiles above sprites. The key is in constants/hardware_constants.asm:

 ; OAM attribute flags
 DEF OAM_TILE_BANK EQU 3
 DEF OAM_OBP_NUM   EQU 4 ; non CGB Mode Only
 DEF OAM_X_FLIP    EQU 5
 DEF OAM_Y_FLIP    EQU 6
 DEF OAM_PRIORITY  EQU 7 ; 0: OBJ above BG, 1: OBJ behind BG (colors 1-3)
 
 ; BG Map attribute flags
 DEF PALETTE_MASK EQU %111
 DEF VRAM_BANK_1  EQU 1 << OAM_TILE_BANK ; $08
 DEF OBP_NUM      EQU 1 << OAM_OBP_NUM   ; $10
 DEF X_FLIP       EQU 1 << OAM_X_FLIP    ; $20
 DEF Y_FLIP       EQU 1 << OAM_Y_FLIP    ; $40
 DEF PRIORITY     EQU 1 << OAM_PRIORITY  ; $80

Every tile on the screen has an attribute byte. The lowest three bits define the color, which is why there's only room for eight colors (from PAL_BG_GRAY, 0, to PAL_BG_TEXT, 7). The other bits control other properties. In particular, the high bit controls tile priority. So if the gfx/tilesets/*_palette_map.asm files could define tiles' priority as well as color, you could make any tile have priority over sprites.

2. Define PAL_BG_PRIORITY_* constants

Edit constants/tileset_constants.asm:

 ; bg palette values (see gfx/tilesets/*_palette_map.asm)
 ; TilesetBGPalette indexes (see gfx/tilesets/bg_tiles.pal)
  	const_def
  	const PAL_BG_GRAY   ; 0
  	const PAL_BG_RED    ; 1
  	const PAL_BG_GREEN  ; 2
  	const PAL_BG_WATER  ; 3
  	const PAL_BG_YELLOW ; 4
  	const PAL_BG_BROWN  ; 5
  	const PAL_BG_ROOF   ; 6
  	const PAL_BG_TEXT   ; 7
 
+	const_next $80
+	const PAL_BG_PRIORITY_GRAY   ; 80
+	const PAL_BG_PRIORITY_RED    ; 81
+	const PAL_BG_PRIORITY_GREEN  ; 82
+	const PAL_BG_PRIORITY_WATER  ; 83
+	const PAL_BG_PRIORITY_YELLOW ; 84
+	const PAL_BG_PRIORITY_BROWN  ; 85
+	const PAL_BG_PRIORITY_ROOF   ; 86
+	const PAL_BG_PRIORITY_TEXT   ; 87

(The exact PAL_BG_PRIORITY_ names are important: Polished Map supports them when editing tilesets.)

But we can't just start using colors like PRIORITY_RED in the tilesets' palette_map.asm files. The tilepal macro packs two tile color definitions into each byte, using four bits per tile: three for the color (PALETTE_MASK), one for the bank (VRAM_BANK_1). So we need to add space for the new priority data.

3. Use one byte per color for tileset palettes

Edit gfx/tileset_palette_maps.asm:

 MACRO tilepal
 ; used in gfx/tilesets/*_palette_map.asm
 ; vram bank, pals
 	DEF x = \1 << OAM_TILE_BANK
-	rept (_NARG - 1) / 2
-		dn (x | PAL_BG_\3), (x | PAL_BG_\2)
-		shift 2
+	rept _NARG - 1
+		db (x | PAL_BG_\2)
+		shift
 	endr
 ENDM

4. Fix the skipped space in palette_map.asm files

The gfx/tilesets/*_palette_map.asm define tile palettes in order: first for tiles $00 to $5F, then for tiles $80 to $DF. Tiles $60 to $7F are skipped because those IDs are used for font graphics. But the skipping is done with a count of bytes, not of colors, so we need to double the counts.

Edit all the gfx/tilesets/*_palette_map.asm files:

-rept 16
+rept 32
  	db $ff
 endr

5. Fix the bank overflow

Now the tileset palette data will take up twice as much space—one byte per tile instead of half a byte—so it won't fit in its ROM bank. Edit main.asm:

-SECTION "bank13", ROMX
+SECTION "Tileset Palettes", ROMX

 INCLUDE "engine/tilesets/map_palettes.asm"
 INCLUDE "gfx/tileset_palette_maps.asm"
+
+
+SECTION "bank13", ROMX
+
 INCLUDE "data/collision_permissions.asm"
 INCLUDE "engine/menus/empty_sram.asm"
 INCLUDE "engine/menus/savemenu_copytilemapatonce.asm"
 INCLUDE "engine/events/checksave.asm"
 INCLUDE "data/maps/scenes.asm"
 INCLUDE "engine/overworld/load_map_part.asm"
 INCLUDE "engine/phone/phonering_copytilemapatonce.asm"

Since we don't specify a bank for "Tileset Palettes" in layout.link, rgblink will place it in any bank that has enough room.

6. Correctly read the extended palette data

Edit engine/tilesets/map_palettes.asm:

 _SwapTextboxPalettes::
 	hlcoord 0, 0
 	decoord 0, 0, wAttrmap
 	ld b, SCREEN_HEIGHT
 .loop
 	push bc
 	ld c, SCREEN_WIDTH
+	call GetBGMapTilePalettes
-.innerloop
-	ld a, [hl]
-	push hl
-	srl a
-	jr c, .UpperNybble
-	ld hl, wTilesetPalettes
-	add [hl]
-	ld l, a
-	ld a, [wTilesetPalettes + 1]
-	adc 0
-	ld h, a
-	ld a, [hl]
-	and $f
-	jr .next
-
-.UpperNybble:
-	ld hl, wTilesetPalettes
-	add [hl]
-	ld l, a
-	ld a, [wTilesetPalettes + 1]
-	adc 0
-	ld h, a
-	ld a, [hl]
-	swap a
-	and $f
-
-.next
-	pop hl
-	ld [de], a
-	res 7, [hl]
-	inc hl
-	inc de
-	dec c
-	jr nz, .innerloop
 	pop bc
 	dec b
 	jr nz, .loop
 	ret

 _ScrollBGMapPalettes::
 	ld hl, wBGMapBuffer
 	ld de, wBGMapPalBuffer
+	; fallthrough
+GetBGMapTilePalettes:
 .loop
 	ld a, [hl]
 	push hl
-	srl a
-	jr c, .UpperNybble
-
-; .LowerNybble
 	ld hl, wTilesetPalettes
 	add [hl]
 	ld l, a
 	ld a, [wTilesetPalettes + 1]
 	adc 0
 	ld h, a
 	ld a, [hl]
-	and $f
-	jr .next
-
-.UpperNybble:
-	ld hl, wTilesetPalettes
-	add [hl]
-	ld l, a
-	ld a, [wTilesetPalettes + 1]
-	adc 0
-	ld h, a
-	ld a, [hl]
-	swap a
-	and $f
-
-.next
 	pop hl
 	ld [de], a
 	res 7, [hl]
 	inc hl
 	inc de
 	dec c
 	jr nz, .loop
 	ret

Notice how SwapTextboxPalettes now reuses the loop it shares with _ScrollBGMapPalettes, and then the whole decision of which nybble to read is no longer necessary because the whole byte defines one tile's attributes.

Anyway—at this point you are done! Now when you edit a palette_map.asm file, you can use the names PRIORITY_GRAY, PRIORITY_BROWN, etc., and the corresponding tile will appear above any NPC.

Screenshot

However, the lightest hue (that's white when you're editing the monochrome tileset PNG) will be transparent. That's how tall grass works: you see only the parts of the player sprite that overlap "white" pixels (actually light green, using the standard outdoor color palette). So design your overhead tiles carefully.