AlmaIF memory regions

[soccercheng]

After tracing the firmware code of AFOCL, I have the following questions about AlmaIF regions.

1. Where do AlmaIF regions reside ? Do they reside locally in AlmaIF ? according to Makefile in AFOCL, it specifies only where instruction/data memory reside by using "generateprocessor ... -f onchip -d onchip ...", what does "onchip" mean ?
2. Why the physical/offset address of control registers is "0x40000000 + j * 0x10000", as described in pocl's XilinxXrtDevice.cc ?
3. Who sets AlmaIF control registers? It seems that pocl almaif device driver does NOT configure control registers? if this is the case, who and when the default value of control registers are configured/set ?
4. The memory mapping of AlmaIF regions from firmware point of view, Why the QUEUE_START is defined as "0" firmware_vecadd.c? It seems that firmware does NOT read AlmaIF control registers accordingly ?
5. Is there any specification of AQLQueueInfo and AQLDispatchPacket ? IT seems the first element of command queue is AQLQueueInfo ?

[TopiLeppanen]

Okay, good questions:

1. The regions are physically in the block RAMs. They are instantiated in the tta_core_toplevel.vhdl.
2. I have done experiments with multiple OpenCL devices on the same FPGA fabric. In that case I separated them by the the 0x10000-offset, and was able to control them in the same AlmaIF-driver instance. The 0x40000000 offset itself was some kind of physical requirement due to the device I was using.
3. The control registers of the AlmaIF interface are implemented by the OpenASIP's "debugger"-component that sits right next to the core. For the implementation of the control registers, see openasip/data/ProGe/debugger/minidebugger.vhdl. Therefore the values of the control registers get "baked in" the synthesized TTA core.
4. QUEUE_START is defined to be zero, as in that design I have separate address space and LSU inside the TTA-core for accessing the command queue memory. I have other designs which do not have a separate LSUs for this, which instead need to add a static offset to the beginning of the queue.
5. AQLQueueInfo does not have specification as it is just the single struct at the beginning of the command queue memory. It takes up as much space as a single dispatch packet just for convenience. Critically it contains the read and write indices to the ring buffer. The actual ring buffer then takes up the rest of the command queue memory. The AQLDispatchPacket is based on HSA's AQL queues command. For more details see http://hsafoundation.com/wp-content/uploads/2021/02/HSA-SysArch-1.01.pdf section 2.9.6
   You probably don't need to read the document to use AlmaIF, and we do not support it in its entirety, but perhaps it is useful background info.

[soccercheng]

Further questions

1. Why not creating discussions board on project AFOCL? cause some topics should be post and discuss there.
2. What exactly the "external-memory": "0x0000,0x10000000" in db.json is for ?
3. How do you know 0x40000000 offset is the physical requirement for U280? which of the essential Xilinx documents describes the requirements? Are there any other regions that can be assigned ?
4. It seems that 0x40000000 overlaps with U55C's HBM[2], according to the dump of "XRT device's memory info" in line 102 of the enclosed log file.
5. In firmware, how do I specify which LSU (load store unit) the memory access goes through ? I should use "address_space" attributes as shown in the following code segment in firmware ? How many address spaces does tcecc support ? what is the mappings between address space # and AlmaIF regions ?

#define __cq__ __attribute__ ((address_space (5)))
#define __buffer__ __attribute__ ((address_space (1)))

1.Do fields in AQLQueueInfo have special meanings ? as we can see that firmware stores information in some of AQLQueueInfo's fields as shown in the code segment below.

queue_info->doorbell_signal_low = DMA1[0];
queue_info->doorbell_signal_high = DMA1[1];

pocl.accel_example.log

[TopiLeppanen]

1. Yes, discussing this in the AFOCL-repo might be optimal, since these are definitely related to that. I opened the discussions board there.

2. It's for the DDR/HBM external memory of the card. I use that field to initialize memory management for that area. Actually for the XRT-based platforms I might have moved away from my own memory management and instead just call the XRT api to manage that for me. IIRC I was having performance issues, when treating the whole external memory as a single self-managed buffer. The field would still be relevant for other platforms, but might not be used for the Alveo platforms.

3. Oh right, now I remember better. The 0x40000000 offset might actually be the offset used for the TTA-core's global memory access. This is related to TTA internals, in which the global memory LSU accesses are either directed outside to the TTA's m_axi-port, or to the internal bram memory (if the pointer is in range of the AlmaIF datamemory). TTA core has to know its own offset, which is given to it as VHDL generic axi_offset_low_g. This part used to make much more sense in the Pynq-Z1-based earlier design, in which the 0x40000000 was also directly the physical pointer that the ARM CPU could access. It still kinda works with the Alveo-based design, but could lead to issues as you point out in Q4

4. Yes, that is an issue I thought of before but haven't yet fixed. I have gotten around it by only using less than 1GiB of external memory. Changing the axi_offset_g_low and axi_offset_g_high generics are the first step to move this offset to somewhere else. I have not done it myself. There's off course still then the issue of only supporting 32-bit addresses inside the TTA core, which I referred to earlier. And this automatic redirecter component I referred to in Q3 can only deal with 32b addresses.

5. Yes, the attributes are exactly the way. Each LSU in the ADF then has one or more address space attributes that must correspond to the ones you use in the firmware. The mapping is actually visible in the #defines you copied. The Command queue region is AS 5, and Data/Buffer/Global (all terms should mean the same thing in this context) memory is AS 1.

6. No, the other fields than read_index and write_index are barely used, maybe the size one is used, but the rest probably aren't. The snippet you've shown is some debug code I was using to take a peek inside the thing running on the FPGA. The struct field names are related to parts of the AQL spec I haven't yet implemented. Probably should've just named them 'reserved'.