In ISE, click on xc6slx45-3fgg484 under Design panel, Hierarchy.
In the window below, expand Design Utilities.
Right-click on Compile HDL Simulation Libraries.
Set target simulator to correct Modelsim version and Verilog.
Set language to Verilog.
Set simulator path to where modelsim.exe has been installed. (ex. "C:\modeltech64_10.0d\win64")
Set compiled library directory (ex. "C:\Xilinx\Compiled") and apply.
Make sure unisim and xilinxcorelib are checked for compilation.
Back in main window, go to Preferences in the Edit menu.
Under ISE General, click on Integrated Tools Options. (ex. "C:\modeltech64_10.0d\win64\modelsim.exe")
Back in main window, go to Design Properties in the Project menu.
Select the same Simulator as above.
Set language to Verilog.
Right-click on Compile HDL Simulation Libraries and Implement Top Module.
Set up ModelSim
From File dropdown menu, change directory to where the Simulation folder should go.
Create a new project. (ex. FPGA1394_QLA_Simulation)
Choose a default library name (ex. "work") and copy settings from modelsim.ini.
Add all .v files in FPGA1394_QLA/Verilog to project, as well as hub_mem_gen.v and pkt_mem_gen.v from ipcore_dir.
This will not compile since it is missing Xilinx libraries that were just compiled. To link to them, create a new library.
Choose "a map to an existing library" and set a name (ex. "unisims_ver"). Set the path to be where the UNISIM library was compiled previously. (ex. "C:\Xilinx\Compiled\unisims_ver")
Repeat and create a map to xinlinxcorelib_ver.
Add glbl.v to project from "$XILINX/verilog/src/glbl.v".
Compile all. This may fail depending on Firmware version since ModelSim has stricter syntax requirements than Xilinx ISE. Double click on the x for the failed files to see error log.
Add a testbench file. For example, see FPGA1394_QLA_tb.v in the Simulation folder.
Under the Project menu, select Add to Project, Simulation Configuration...
In Design Unit(s), add the testbench you'd like to simulate and the glbl file. (ex. work.FPGA1394_QLA_tb work.glbl)
Uncheck Enable optimization.
In Libraries tab, add unisims_ver and xilinxcorelib_ver.