System DT Meeting Notes 2020 - OpenAMP/open-amp GitHub Wiki

Table of Contents

2020-12-03

Attended

  • Grant Likely - Arm
  • Krzysztof Kepa - GE
  • Bill Mills – Linaro
  • Rob Herring – Arm
  • Joakim – Linaro
  • Mathieu Poirier - Linaro
  • Arnaud – ST
  • Loic – ST
  • Sumam Anna - TI
  • Mark Dapoz – Wind River
  • Tomas – Xilinx
  • Stefano – Xilinx
  • Ben – Xilinx
  • Bruce – Xilinx
  • Ed – Xilinx
  • Hannes Tschofenig - Arm

Agenda

  • Stefano, Loic, Rob: Continue discussion on bus firewalls in the context of system block diagrams
  • Stefano: Remote processor bindings

Action items

  • Stefano: Write-up and continue discussion on list

Notes

  • Recording: (I am not sure how long before the recordings expire or I will hit my storage limit, so if you need to catch up by watching the recording, please download it in the next couple weeks) Recording link
  • Firewalls:
    • Stefano showed diagram that show how firewalls are spread all over and are somewhat different than interconnects. We could potentially still use interconnects to describe firewalls.
    • Loic described the ST implementations, which are similar to Xilinx.
    • Stefano showed three ways of describing firewalls:
      • 1. Original proposal introducing new bindings for firewalls
      • 2. Initial interconnects proposal, introducing two new interconnect properties
      • 3. New interconnects proposal adding interconnect-names (might be misusing the interconnects specification)
    • Rob: Is there a 1-1 relationship between interconnects and firewalls?
      • Stefano, Rob, Loic and Bill discussed what info is really needed. Feeling is that there is still info missing.
      • There doesn't seem to be a 1-1 relationship between interconnects and firewalls. Firewalls are basically "System MPUs".
      • The master side is similar to an IOMMU without the translation portion, and could potentially be described using IOMMU bindings. But we have no way to describe the slave side.
    • Discussion with TI on whether the proposed firewall/interconnect bindings could be useful for them too.
    • Certainly in system device tree. At runtime (regular device tree) is more complicated because typically a firmware interface is used to change firewall settings (e.g. SCMI or something else.)
    • Stefano: the firewall description in device tree could still be useful for the firmware itself. However today that class of firmware typically doesn't use device tree at runtime, only at build time like Zephyr.
  • Next: Write-up and continue discussion on list

2020-10-22

Attended:

  • Bill Mills - Linaro
  • Poonam - NXP
  • Stefano Stabellini - Xilinx
  • Joakim Bech - Linaro
  • Ben Levinsky - Xilinx
  • Ed Mooring - Xilinx
  • Mathieu Poirier - Linaro
  • Tomas Evensen - Xilinx
  • Nathalie Chan King Choy - Xilinx
  • Bruce Ashfield - Xilinx
  • Arnaud Pouliquen - ST
  • Etsam Anjum - MGC
  • Rob Herring - Arm
  • Suman Anna - TI
  • Krzysztof Kepa - GE Research
  • Loic Pallardy - ST
  • Grant Likely - Arm
  • Frank Rowand - Sony

Agenda:

  • Bus firewalls & interconnects
  • Remote processor bindings

Action items:

  • Stefano & Loic: send the block diagrams of ST & Xilinx systems
  • Stefano: Kick off discussion on aligning the remote processor bindings
  • Stefano to send out slides & examples (DONE)

Notes:

  • Recording: (I am not sure how long before the recordings expire or I will hit my storage limit, so if you need to catch up by watching the recording, please download it in the next couple weeks) Recording link
  • Stefano's slides and example: He sent it to the mailing list after the call: Message link
  • Bus firewall controllers example
    • Amba_xppu + 4 devices under amba
    • Ethernet: Bus master ID: ID at each firewall controller
      • # is same b/c on Xilinx HW these # are global but might not be same on all platforms
      • Lpd_xppu can protect ethernet from foreign access
  • Interconnect bindings
    • Introduce interconnect controller
    • Interconnects property under sdhci: Linking to the interconnect controller & passing additional info
    • Firewall controller is similar to interconnect controller & additional info
    • Kind of similar
    • Important differences (not saying it's a bad idea to use interconnect bindings for firewalls)
    • Interconnect & firewall controllers are 2 different blocks in Xilinx & ST HW
      • Might want to configure them at different times
      • Info might come from different DTSI files
      • For interconnect: QoS info
      • For firewalls: Blocking or not blocking
      • Rob: While it's separate controllers, the topology of the system would be the same? You wouldn’t' have distributed interconnect controllers & single firewall controller
        • Stefano: Possible to have opposite: More firewall controllers than interconnect controllers
        • Loic: Topology of both could be different. Several interconnects between 1 master & peripheral but only 1 firewall. Agree that we have only 1 view of list of masters & list of slaves + properties we have to assign to each port.
      • Rob: Concern is that last bullet talks about how we represent master & slave. Slave side is easy b/c device tree is representing the slave side, so if it doesn't show all the steps, it's missing from the tree. Interconnect was supposed to represent the master side.
        • Stefano: Good to hear b/c that's what we tried to do
  • Mapped firewall controller to interconnect controller
    • Loic: Agree that DT representation of all slaves that 1 master can access. But, we aren't necessarily talking CPU master. Could be any master, like Ethernet or PCI controller. Can do bus txns like CPU, but they are slaves programmed by CPU. Have slave port for their programming & 1 master port to execute. That's why we need to describe them as both slave & master.
      • Rob: Good distinction
    • Stefano presented a hybrid distinction. Note that this is a 1st attempt, but gives you an idea of what could be done. Abstract from the details & see if solution could be acceptable.
    • Master side doesn't reuse interconnect as-is without -master part of the name lets us keep QoS & firewall data separate.
    • Interconnect controller to describe firewall controller
    • On device side, have both interconnect master & interconnect slave to describe the slave port, like what Loic highlighted
    • Poonam: Firewalls are isolation units? We get programmed w/ master ID & slave ID & expect these properties would get used & the HW programmed.
      • Stefano: Yes. These properties should be able to describe enough that SW or FW could program.
      • Loic: 2 needs for such properties
        • System DT description: How to assign peripherals to masters & program right firewall controller w/ right rules (thanks to phandles can know where you have to define access right control). System DT config generation.
        • In ST, we propose to keep interconnect slaves in Linux kernel DT to verify that Linux kernel can access the peripherals. Firewall may be accessible in read, not write, for everybody, so can check. Can modify @ runtime the DT. Can avoid crash at kernel when try to access peripherals. Make sure customer can have stable platform & not try to access peripherals not assigned to Linux.
    • Poonam: Who programs the HW if read only from OS
      • Loic: Table from firewall is under control of the security master of your system (e.g. OP-TEE)
    • Poonam: How will it know this configuration?
      • Stefano: Orthogonal to firewalls. System DT has domains, at build time we will generate the actual config (binary table) of the firewall controller. Probably will process w/ Lopper plug-in ("lops" = python modules to do additional operations). Can read in info & generate vendor-specific table for your bus firewall controller.
    • Poonam: Could be blob?
      • Stefano: Trying not to have blob so can read & modify domains by hand, without having to look at the binary description. This way it's very transparent, easy to read, easy to write. Hard part is generated by tool.
    • Back to HW description of Interconnect Bindings for Bus Firewalls
      • Using interconnect controllers as bus firewall controllers
      • Interconnect master & interconnect slave new properties (could pick better name)
        • Link to controller
        • Specify info like bus mastering ID
        • Who are the bus firewall controllers protecting
    • Stefano: Does this look like reasonable approach?
      • Rob: Bothers me that we're pointing to the same node multiple times. e.g. Ethernet node pointing to lpd 3 different times. Seems duplicated.
      • Loic: ST, you'd have 3 different phandles b/c have own interconnect w/ different
      • Stefano: This interconnect for QoS would also be pointing to something else. None of the Xilinx bus firewalls do QoS. You can forget the 1st one. Sometimes we will have same on master & slave side.
      • Rob: What interconnect points to doesn't have to be QoS. Describes a path from device to somewhere else (usually memory) and the cells meaning is up to interconnect controller. Could
      • Loic: Even if we have 3 on same line after interconnects - how to get the info that the 2nd one is slave & 1st is master definition. B/c they are all optional. If we can set some names, like GPIO, it's possible.
      • Rob: Think we have that now
      • Loic: Was thinking fixing names is more difficult
      • Stefano: More difficult to parse
    • Stefano: Revised example, removed 1st line & if we put rest on 1 line.
      • How to say ethernet is also master & also slave & would need to add another property
      • Would have to make property standard for all controllers, or how do you know how to identify slave (regardless of HW) - always same cell.
      • Rob: Interconnects is a bit odd b/c have 2 phandles & cells b/c describing a path. The cells have no meaning to the node they are in. They have meaning to the phandle they point to & that defines # cells. Index has meaning to the local node.
        • e.g. interrupts: Transmit & receive interrupt for a device. Which is 0 & 1 is meaningful to device but not to interrupt controller. Cells defined has meaning to interrupt controller & not the device.
      • Stefano: Position?
      • Rob: There's different indexes in here (interconnects is a little different). Each phandle defines new entry & cell 0, 1, 2 has meaning to provider (lpd_xppu).
      • Stefano: So, we could we mark clearly which is slave?
      • Loic: Why only mention lpd_xppu once? Why not mention only the slaves, then definition for master, then elsewhere definition for interconnect - not all in 1 list.
      • Rob: I only said it's a concern b/c looks like we're duplicating info
      • Loic: Yes, have same controller, but should allow for several phandles. If it's same controller, need to specify in your argument, the master properties then the slave properties.
        • In ST case would specify twice b/c not same IP for master & slave
        • If you can specify master & slave in same controller, then __
        • 1st put mandatory options, then optional ones
      • Loic: Could we have list for slave & masters after LPD
        • Stefano: Problem is that any other would now have to have 4 cells. What if you're only interested in the master properties.
        • Loic: Are you sure?
        • Stefano: Each controller can have different # of cells parameter, doesn't depend on device side.
          • Example: lpd_xppu can block things & protect regions, so need 4 parameters in both cases
      • In Xilinx may have 1 slave & multiple masters & ST may have more masters & 1 slave
    • Loic: For clocks, we're repeating phandles in list.
    • Rob: Would be helpful to have block diagrams to show what the HW looks like, so can understand the differences between the different SoCs
      • Loic: ST, Xilinx, TI, NXP have very vendor-specific. That's why looking to offer more open possibilities for System DT to be open to all design.
    • Rob: Maybe having everything in interconnects doesn’t make sense
      • Loic: If want to put under interconnect property, need to fix some names w/ fixed meanings. Like default & GPIO for pin control.
      • Rob: Right
      • Stefano: Fixed names or different properties w/ different names achieves the goals of separation & clarity
      • Rob: We can do where it makes sense. If you look @ GPIO & interrupt flags we have standardized b/c after years they're all the same & didn't start out that way & could have one that’s completely different in terms of what cells mean
    • Stefano: Good to know we have several options that look acceptable
    • Next steps:
      • Stefano & Loic: send the block diagrams of ST & Xilinx systems
  • Remote processor bindings
    • Stefano had slides of the strategy, without the binding details
    • Would like single place for all the info & minimal info
    • Want to keep current DT bindings & regular lifecycle + single simpler DT description. Just enough info to make it work. Vendor specific info only one 32-bit field.
    • Long-term, would be good to have more alignment between vendors, but it's not required now
    • Each vendor could have Lopper plug in & use that to generate the traditional bindings.
    • So, we can make changes to the traditional bindings, but we're not forced to do so
    • Acknowledge that the remoterpoc bindings are fragmented & should align long-term
    • Showed example of Xilinx DT binding for remoteproc (Ben is upstreaming & Rob is reviewing)
      • Moved out the TCM nodes & now much more aligned w/ System DT
      • Resource group to share resources across domains
      • Only vendor-specific info: 0x13 gets translated by Lopper & gets translated to 2 lines
      • Would like to be able to take a single input & one bitfield, generate the traditional remoteproc description that works with Linux to retain compatibility with Linux kernel releases
    • Rob: Concern is that "Lopper will handle it" is hand-wavy. If you keep adding to it, you will have problem that Lopper plug-ins are too complex.
      • Stefano: Fair. If remote proc bindings were more aligned, we wouldn't need Lopper plug-ins. This is to retain compatibility with what we have now.
      • Ben: Earlier version have complete working implementation of Lopper generating DT that requires only small changes to work with upstream proposal.
      • Stefano: As upstream
      • Stefano: LOC?
      • Ben: Python wrapper can be stripped down to maybe ~100 lines with half being boiler-plate.
    • Rob: Concern: Both TI & Xilinx are defining cortex R5 remote proc interfaces & no alignment besides using remoteproc.
      • TCMs is another aspect. Need a node that could be represented in System DT as you have it here, then translation of address is separate for system DT. Then, how to represent in remoteproc should be same
      • Stefano: Agree. Can have discussion on aligning the remoteproc bindings.

2020-08-31

Attended

  • Krzysztof Kepa – GE
  • Mathieu Poirier – Linaro
  • Etsam Anjum – Mentor
  • Anrnoud Pouliquen – ST
  • Loic Pallardy - ST
  • Suman Anna - TI
  • Dan Milea – Wind River
  • Mark Dapoz – Wind River
  • Tomas Evensen – Xilinx
  • Stefano Stabellini – Xilinx
  • Bruce Ashfield – Xilinx
  • Ed Mooring – Xilinx
  • Ben Levinsky – Xilinx

Agenda

  • Remote processor bindings

Action items

  • Stefano to send out examples and slides.

Notes

  • Stefano went through slides.
  • Overall idea:
    • 1. Specify remoteproc channels with minimal information in System-DT in a way that is as common as possible for all vendors. In particular, we are avoiding to have to specify the same memory regions in more than one place. A resource group is specially marked to contain most information.
    • 2. Use Lopper to create the vendor specific remoteproc specification. Generating reserved-memory, etc. information that has duplicate information.
  • Discussion about how TI and ST might generate their specific information from this specification.
  • Stefano to send out examples and slides
  • Ben: Xilinx backend to Lopper is upstreamed and available as an example. Might change as upstreaming continues.
  • Question: How do we configure VirtIO?

2020-Jul-20

Attended

  • CVS
  • Dan Milea
  • Eric Finco (ST)
  • Etsam Anjum
  • Bruce Ashfield
  • Ilias
  • Josh Pincus
  • Nicolas Saenz Julienne
  • Loic Pallardy (ST)
  • Grant Likely
  • Tomas
  • Rob Herring
  • Kevin Chappuis
  • Nathalie Chan King Choy
  • Stefano Stabellini
  • Ed Mooring

Agenda

Stefano to present unified proposal based on ST and Xilinx proposals, along with example

Action items

  • Stefano to share slides
  • Xilinx and ST to discuss interconnect binding, considering that it could be expanded beyond QoS, and review Rob's earlier feedback to Benjamin on pin control
  • Stefano to check if configuration interface assumes that IDs are global
  • Stefano to work with Bruce to prototype something w/ Lopper to generate a bus firewall configuration table
  • Stefano to present openamp-remoteproc binding for System DT at next call

Notes

  • This proposal describes bus firewalls that includes HW description of firewall controller, as well as configuration part in System DT
    • ST & Xilinx proposals were very complementary
  • Might have multiple Firewall controllers
  • Need a new ID space to say what's blocked & what's allowed
  • Firewall configuration in System DT is older part of proposal
  • Firewall Controllers
    • What to protect who
    • Came from ST proposal (Benjamin's patches to LKML)
    • For Xilinx, firewall-cells will be 0, but it's important b/c some vendor specific info might need to be passed
    • Firewall-0 property: Which controller should be configured to protect this device to prevent accesses. 1, 2 are additional info that you might need to program to protect ctrl0
    • Loic: We can have list of firewalls
      • e.g. memory controller (serial NOR)
        • Can have memory map view of serial NOR
        • May have firewall to protect access to controller, one to protect access to memory map address space
  • Bus Master ID
    • Need to be able to identify the thing you want to block
    • ID of the firewall space
    • Has its own device ID (bus master ID)
    • Txns get marked by bus master ID
    • Doesn't need to be global. Could be per controller. -> controller phandle + number
    • On Xilinx boards, they are global (so all IDs are same), but this gives us the flexibility
    • Rob: We already have the interconnect binding that deals with bus master side of SoC. This would need to fit into that somehow.
    • Loic: Properties to define bus ID for any master?
    • Rob: Binding called interconnects for describing bus hierarchy on DMA side
    • Loic: e.g. PCIe, Ethernet, USB, which are masters on the bus. We can assign them specific ID. e.g. processor non-secure context will have same ID as CPU, but if need to isolate completely from PCIe controller b/c sharing DDR, can allocate specific bus master. Notion of programming the master ID for 1 master
    • Stefano: CPU can have bus master ID property too
    • Rob: Why wouldn't it be firewall-cell
    • Loic: Could, but we suppose that at one moment we need to know where to get our ID. e.g. at runtime, if need to check access rights, need to know where our ID is programmed
    • Stefano: There's what you need to protect & what you need to protect from.
    • Rob: You could define a single property w/ 3 entries. First two are for bus master side & 3rd for slave side
    • Loic: Might not be in the same controller
    • Rob: Doesn't matter. Device w/ 2 interrupts could go to 2 different interrupt controllers.
    • Stefano: Given that the 2 functionalities are so different, it's easier to understand if they are separate
    • Rob: Convince me that they are that different.
    • Stefano: Will look up interconnects to understand that binding
    • Rob: If someone hasn't already put interconnects in CPUs, that's coming
    • Stefano: Reason why really different, there are peripherals that are only slaves or only masters or both from firewall perspective. It's really easy to mix them up. For clarity, it would be very good to clearly marked as separate, whether in multiple or single property.
    • Rob: You're having 2 different properties referencing the provider
    • Stefano: Do you have suggestion to make it follow 1 pattern
    • Rob: Combine it into 1 property. What you've copied from is pin control binding
    • Loic: If need to dynamically change the firewall. Lopper use case if you want to release or not.
    • Rob: Explain why you want to do that & justify it.
    • Loic: About interconnect src & destination?
    • Rob: It's getting expanded
    • Loic: Indeed, Stefano, we need to look. Maybe firewall could be described as a bus in that way. Let's see if can use the same descriptions.
  • Configuration
    • Won't cover details of this b/c discussed it previous call
    • Only change is renaming firewall to firewallconf
    • To provide initial config of firewall
    • New concept: Resource group (e.g. domain@0, domain@1), can group together resources & share them across a domain
    • Firewallconf property
      • Links to a domain or device
      • Is it blocked or allowed & with what priority (block, allow, block-desirable to block if you can w/ priority #)
    • Firewallconf-default property
      • So you can block everything easily
  • Full example
    • HW description same as before. 2 firewall controllers. CPU controllers w/ bus master IDs.
    • Firewall-id
    • Bus-master-id if bus master
    • Running on cortex A cluster, other on Cortex R cluster
    • For the 2 shared resources, everything is blocked by default, except there is block-desirable 8
    • Lopper can retrieve the bus master IDs to write the config of the system
  • Will work with Bruce to prototype something w/ Lopper to generate a bus firewall configuration table
    • Rob: Are the bus master IDs defined in HW in global # space?
    • Stefano: Vendor specific. Yes for Xilinx.
    • Rob: Configuration interface assumes that they are.
    • Stefano: Idea is that the configuration space shouldn't assume that they are. Worth double-checking.
    • Loic: Bus master ID is partially fixed for ST. Fixed for CPU. Programmable for other masters. This programming should be done by firmware at boot.
    • e.g. protect can from mmc0. Look at lpd ID. Retrieve the # and put in rules of table.
  • Loic: Summarizing, firewall config default is for System DT and to build config table for each firewall that would be programmed by firmware before you boot. This property would never appear in U-Boot and kernel device tree.
    • Stefano: Yes, we would use at build time, before boot time
  • Loic: Interconnect is more for QoS & bandwidth. For us, need access right
    • Rob: That could be expanded
  • Ilias: Initial config would go away & not be exposed to kernel. Initial discussion on DTE is basic DT exposed to all the guests & bits of firmware will add what they want to configure
    • Loic: Firewall config will be used by System DT & tools like Lopper to build config table for firewall. Will have list of all the config to set, entry by entry. Will be able to use firewall phandle or interconnect phandle to check your config if you can access a peripheral or not. You will keep the config table in the firewall node & set for the security only once. Firewall config info here is only needed by the System DT to create these tables.
    • Ilias: OK, sounds fine
    • Stefano: There are many steps before the boot time, this is where it's useful
    • Loic: e.g. Firewall init table not discussed yet w/ Stefano. Imagine 1st basic init w/ TF-A, then full table added & managed by OP-TEE and be able to overwrite properties for 1 existing node
    • Ilias: Let's discuss more on Wednesday
    • Loic: Stefano, can you come back on the domain node. It's mainly for system DT & this won't be used at runtime. More for the tools to know how to create the different layers for DT from boot until kernel one.
  • Stefano: Have been working on openamp-remoteproc binding for System DT. Would like to show at next call
  • Tomas: Once Lopper produces, will start to make easier to see when it's system DT & when it's DT for Linux.

2020-May-13

Attended

  • broonie
  • Stefano Stabellini
  • Nathalie Chan King Choy
  • cvs
  • Benjamin Gaignard (ST)
  • Ed Mooring
  • Tomas Evensen
  • Bruce Ashfield
  • Rob Herring
  • Ilias
  • Joakim Bech
  • Loic Pallardy (ST)
  • Mark Dapoz
  • Krzysztof Kepa

Agenda

  • Stefano to present proposal for bus-firewall configuration in system device tree

Action Items

  • Nathalie sync up w/ Francois about who should be invited to System DT call (some individuals at DTE call didn't get this invitation)
  • Stefano to go back & discuss w/ Xilinx XMPU expert.
  • Stefano: Revisit wording of IDs
  • Stefano & Tomas to sync-up with Loic & Benjamin. Target for next System DT call to discuss how the 2 proposals combine. Include description of the use cases you're giving solution for.

Notes

  • TO DO: Link to Stefano's slides
  • Resource Groups will become more important
    • Collection of devices accessible by 1 or more domains
    • Change we made: Use different property name. To avoid confusion & b/c definition different, used "include" instead of "access"
    • What if you have device you want to share across multiple domains?
      • Using resource group for sharing, it becomes clear which resources are shared across domains & easier to check
    • Also will use for bus firewalls
  • Bus Firewalls
    • Needs info in list on slides
    • Stream ID range -> IDs is an open question, but assume Stream ID for this discussion
      • Other bus firewalls might have different ID space
    • address range to block/allow access
    • some may have allow rules & block rules
    • priority is not required, but often bus firewall has only few slots. So, would be good for tool to know which rules will fit & which will not fit. Make rule that higher priority rule goes in & others as best effort.
    • Benjamin: How will it work between 2 address blocks?
      • Stefano: Think example will answer the Q
    • Target Example: 2 domains blocking each other's access to their resources
      • CPU cluster of domain 0 is blocked from accessing memory regions & region of device
    • Benjamin: phandle on device. Here seem to have double things.
      • Stefano: The rest of slides might provide some clarity
    • Tomas: Could be DDR memory, or address-mapped control registers. As long as it's addressable, it's the same. Even if that's controlling an Ethernet, for e.g.
    • Rule #1: Address ranges
      • In this slide, wherever it says "memory range", should say "address range", ref. Tomas' comment above.
      • Allows us to avoid writing all the address ranges we are protecting, so reduces some duplication
    • Rule #2: phandles
      • Using links instead of stream IDs hard coded in the attribute
      • Makes easier to check for correctness
    • Rule #3: firewall-default
      • When you want to block everyone except small set
    • Rule #4: "Stream ID Self" is always allowed
      • There is no useful scenario when you want to block yourself from your resources
    • Example #1: Same example we saw before, but now we can go in more detail
    • Example #2
      • Domain 0 & Domain 1 block everyone else
      • Highest priority is to protect the resource-group
      • Domain 0: All stream ID except your own stream ID are blocked from accessing the memory range & MMIO region of MMC0
      • Resource group is blocking everyone else w/ higher priority. All stream ID except for domain 0 & domain 1 are blocked.
    • Example #3: 1 domain has higher priority
      • Everyone is blocking all stream ID
      • Domain 0 blocking at higher priority
      • Domain 1 & Domain 2 are allowing Domain 0 to access
    • Benjamin: ST propose to set firewall depending on node more like pin control configuration. This is quite different POV.
      • Tomas: On each node ST says to set each firewall. For us, we see it more of a SW configuration than a HW description, so you can change it without changing the DT that describes the HW. So, we put these in different places. That's a goal for us. Also, we want to express things as "I have resources here, what do I want to protect from" b/c think that's how SW & Safety folks think. The other way would be to say each master doesn't affect memory or a device or someone else, but then if you do it in 1 place, you don't have the security or safety around it. So, if someone adds a new Ethernet controller somewhere, you don't have to remember to say also this one doesn't mess with my region.
        • Keep HW & SW config separate
        • Describe what you want to protect
    • Benjamin: We don't have this problem of everything is allowed & need to block everybody or open to some groups.
      • Loic: Maybe not 100% opposite. We can verify at runtime that we can access the resource. So, we know where to look in firewall config so we know where to access it. If we can't, we can request access via secure monitor, that will open firewall for the SW. Here, it's more initial firewall config that you want to describe. Now I understand the different domains, but I don't understand the priorities. For me, you have access or don't.
      • Stefano: In the firewall, you do/don't have access. Priority is to simplify the writing of the rules for someone who wants to protect more than they can protect. If can't protect the priority 9, will throw an error. So, you can write the rules so it matches what does/doesn't get protected.
      • Loic: If you put levels of priority, you will add much complexity in tools to protect something. You may change the firewall config & SW was working fine as priority 6, but then you lose access to the resource b/c you added something w/ priority 7. We should not jeopardize SW execution.
      • Stefano: Agree w/ sentiment that priority field would simplify the spec & simplify understanding from SW. Reason we might have to have it (maybe optional) but the HW firewall on Xilinx board is taking complex configuration. So, the tool might have to know the priority to play some complex tricks to generate firewall rule & not simple 1:1 translation.
      • Loic: So, you have certain amt of entries & can't protect everything.
      • Stefano: Additionally, our firewall rules have address masks & need to group together >1 address range. Because of this, the priority info would be useful for Lopper to be able to generate the complex set of rules in the language the firewall understands. If you don't want to use priority, just put 9 for everything. Stefano to go back & discuss w/ Xilinx XMPU expert.
      • Tomas: Could say block, allow, desirable block
      • Loic: Yes, something like that.
      • Tomas: Could create group for these things to be protected
      • Loic: So in your domain you put what MUST be protected & may be protected & not protected
      • Stefano: Yes, Example #2
      • Benjamin: So it means not all the rules may be applied b/c no more space in firewall?
      • Stefano: The idea is that highest priority must be applied (9 in this example), but like the feedback of block, allow, block-desirable, allow-desirable. Will go back & try to clarify.
      • Mark: How would user know if it's protected if it's desirable? It may change
      • Stefano: SW will only get description of what it should access. We are generating a traditional DT out of this.
      • Loic: Think this is where is complementary w/ Benjamin proposal. At runtime, can check b4 accessing peripheral that we have access for it. Important b/c w/ system DT you want to link firewall config w/ DT node enabling & broadcasting info for all the SW running on the platform. But, you know it was working for customer & customer can update kernel & DT with/without changing the firewall. Where HW is not safe, you will catch & not boot kernel & put message on console. Benjamin has done proposal on list: Before probing drivers check that it's allowed & that way know that firewall is well configured & aligned w/ DT.
      • Tomas: I'm hearing that 1 view of firewall is high-level initial firewall (what we have here). You guys need lower-level firewall at runtime.
      • Loic: Yes, this is why both proposals can fit together. We need way to put together initial firewall. ST has multiple firewalls. We should, thanks to this list, be able to generate firewalls for DDR, internal RAM, etc. Today, we don't have generation from 1 to DTS/DTB & customer may update over the air just a part of system. Need to be able to boot in safe way U-Boot & kernel to check that it's OK.
      • Tomas: Runtime verification is really important. We discussed in a previous call, if you have a description like this, how you describe your firewall will be vendor specific. We will have a back-end to Lopper specific to us. ST might want to have a Lopper back-end that adds per-device attributes.
      • Loic: we put link where to check in HW. It's fixed. Like interrupts, how they are fixed. CAN you know is protected by this part of firewall & so you can check it's well configured there. Here it's more dynamic for initial config of firewall that we'll generate. In Lopper, will have different generation of firewall for NXP, Xilinx, ST, etc.
      • Tomas: Yes. We are trying to see if there is a way at higher level for user who moves from ST, NXP, Xilinx part doesn't have to do it in a different way.
      • Stefano: Or at least can use same flow to generate firewall rules
  • Open issue: ID Space
    • Stefano: Xilinx firewall takes IDs that are effectively Stream IDs (IO MMU ID). What do you use?
    • Loic: We can identify all the masters
    • Stefano: I'll have to be careful how I word it so each vendor can have its own ID
    • Loic: Example: PCIe master on bus w/ lot of security issues. When we are on open platforms, each CPU on our SoC has own ID, but masters like PCIe can select the ID. If I don't have close & secure platform, I can have Cortex-A controlling PCIe & PCIe can access what Cortex-A can access. If I want to secure the platform & create sandbox for PCIe, I can dynamically change ID of the PCIe. PCIe will need its own domain, sandbox, memory. Have this on set-top-box chipset b/c 4K programs & encrypted video, have to close everything to ensure the key will never be sent & can't attack the SoC. Need to restrict access of PCIe card.
    • Stefano: PCIe is dynamic & more complex
    • Loic: Can start by doing it static. If we want to isolate PCIe, can do it from beginning. Customer may want
    • Stefano: What is name of property you use? master ID?
    • Loic: can call it Master ID
    • Stefano: Good way
    • Rob: IDs are usually HW-specific.
    • Loic: It's a bus ID, so ID for translations sent by 1 master
    • Rob: Ultimately, you want ID tied to HW somehow. IOMMU: Stream ID + some other info. Want to avoid making up your own #s.
    • Stefano Could say they are bus mastering IDs to avoid making up #
    • Rob: Clock IDs are usually just made u, but sometimes correspond to register offsets. To back up a bit: Where is the description of the HW in a not DT terms, b/c you're giving me the solution, but it's hard to see the problem. Can you express it as "request access from OS", "boot time configuration". Can you enumerate the use cases & requirements.
    • Tomas: In our case, the problem statement is firmware during boot. Get master IDs & address ranges to protect. HW registers to be programmed by the firmware that's running on a special processor.
    • Rob: I expect most implementations will be firmware based for all the resources. I'm getting patches on list from ST & proposals on this call from Xilinx, and discussion on the call. Bit of disconnect. Would be good if you could work together to come up w/ 1 solution, or why there should be 2 solutions.
    • Tomas: We should get together w/ ST
    • Loic: We need to align on vocabulary & come back on execution domains to see what is an execution domain (not just SW, U-Boot, kernel, Xen) but also master & mode (secure/non-secure). So, we can see where ID coming & is it enough or if it needs to be associated to something else.
    • Stefano: Can put the 2 proposals side-by-side & see where they are aligned/not
    • Tomas: Assumption has been that this will be pre-processed to get to firmware. This info by itself will not be there at runtime, to set up the initial firewalls.

2020-Apr-06

Attended

  • Grant Likely
  • Tomas Evensen
  • Arnd Bergmann
  • Benjamin Gaignard (ST)
  • Loic Pallardy (ST)
  • Dan Milea
  • Etsam Anjum
  • Frank Rowand
  • Joakim Bech
  • Bill Fletcher
  • Ed Mooring
  • Francois Ozog
  • Mark Dapoz
  • Mpujol
  • Rob Herring
  • Stefano Stabellini
  • Nathalie Chan King Choy
  • Bruce Ashfield
  • Kevin Chappuis

Agenda

  • Follow up from the System DT FAQ talk during Linaro Tech Days
  • Firewall
  • Lopper
  • Continue to discuss configuration lifecycle

Action items

  • Stefano: Will think through if can provide what you need in execution domains for runtime. This example more focused on boot time.
  • Bruce: Send out info on Lopper once it's posted on GitHub

Notes

  • Linaro Tech Days System DT Presentation
    • Who attended? 8 of the attendees on this call
    • Summarizing what was covered in that talk
      • What is System DT?
      • Domains -> More specific: Execution domains
      • HW description & configuration
      • Example shows all 3 concepts that we've already seen
      • Default
        • We discussed it before, but never summarized it.
        • Why? Both convenient & common. Also turns system DT to a smoother addition to DT spec, less revolutionary & more of an addition than changing.
      • Interrupt controllers
      • Instead of putting under "chosen", put under its own top-level node "domains"
        • An execution domain might have boot args & need its own chosen node & would be confusing to have nested chosen
    • Grant: Like the way that "domains" is a common container point for all the stuff in the execution domain
    • Rob: Looks like what I was proposing where we move everything down a level. Difference is CPU R5 is a phandle here, which would be at the top level.
    • Stefano: Yes, CPU cluster containing the 2 R5s.
    • Rob: Why have it at top level? Why not in the domain?
    • Stefano: Similar to your proposal b/c your question about chosen reserved memory made me re-examine this. Want to have distinction between description & configuration - it's in a way static, it's shipped w/ the HW. Execution domain concept is not just where it's running, but what memory, what devices are assigned & how.
    • Tomas: R5: Want to describe physical attributes of the 2 R5s. Could configure as 2 domains of R5s or lockstep R5. Want to have distinction between how you specify the HW properties & how they are configured.
    • Rob: Works fine on CPUs b/c they don't have any addressing in the sense of the root of the DT is the view of the address space. But when you have nodes where there is addressing, then you're back to the same problem where root node has single fundamental view of address space. E.g. Regular memory nodes have an address
    • Stefano: Idea is to be configured as a range. Memory attribute under CPU just for memory ranges.
    • Rob: Say you have a device at address 1000. For A cores, that's 1 device. For R cores that's some other device or memory. If you want to describe the HW at the root level. You have 2 things at the same address.
    • Stefano: If we have 2 different devices that are at same address of each CPU. From the HW perspective they would be described w/ same trick used for the interrupt controllers: Indirect bus & address map. Refer to the "Interrupts" slide in presentation. Then at execution domain level, you 1st link to the CPU (one of the 2 R5s or both), then view of the system becomes the view of that CPU cluster & it flows from there which devices are visible at which address. Lopper should be able to check (in future).
    • Rob: Indirect bus definition is allowed but not valid & hopefully someday we'll check for this.
    • Rob: The root node is 1 view & the only way around that is to add 1 level.
    • Grant: I agree, if bus is not on the root, they can't have reg property & ranges property. These node names are invalid. If they were inaccessible buses, could do without addresses attached.
    • Rob: But that gives new problem: How do you have more than one?
    • Grant: So would have to give a specific name instead of a generic name.
    • Stefano: Can we still have sub-node w/ address?
    • Grant: Yes, but you wouldn't have to @address on the bus node.
    • Stefano: It makes sense b/c doesn't automatically translate to parent address node.
    • Grant: I can't recall if you need a ranges property to make it all work.
    • Rob: You'd need ranges to be translatable
    • Grant: This would be start of translation domain. You probably don't need ranges… might be handled by indirection from execution domain… That shouldn't be too massive of an impact on what has been done. Should be fairly straightforward.
    • Tomas: Good feedback
  • Bus Firewalls
    • Stefano:
      • We understand the question a little better now.
      • Lopper would look at System DT & execution domain config & have plug in to generate config needed by the bus firewall. The info is all there, just not in the format that the bus firewall needs.
      • Issue could be too much info
    • Tomas:
      • 1 way to do it is to have a way to say: protect this thing from this master & what should not be protected
      • From usability POV, you want to protect a domain (its memory, devices, registers)
      • So, the way to specify what should be protected is through domain concept
      • But then there's problem that Stefano will talk about next
    • Frank: Has decision been to have system DT include config or still cautious?
    • Grant: Want to avoid too much data in DT, but system is only valid if we know how system is used. Collecting info of how it fits together in domain node seems appropriate to me (gut feel, can't really do strict policy here)
    • Frank: We have to be explicit on if we think it's a valid configuration & explicitly justify it
    • Tomas: That's why we're really trying to distinguish the config info & put it all in one place in "domains"
    • Frank: Let's specify that domains
    • Benjamin: Config is owned by device node. We are more in view to do like pin controller configuration b/c main point for us is to change configuration at one time & switch device from Cortex-A to cortex-M. That's what I've proposed on mailing list. Been working on firewall bus. Will send out next week.
    • Frank: Interesting if adding/removing devices from system view when system is up. See that with overlays & FPGA is the main user of FPGAs & they are tiptoeing thru it carefully. You have to be super careful.
    • Rob: I think intent is to do it at boot time, instead of when running?
    • Benjamin: Let Linux assign frame buffer over to co-processor
    • Tomas: We have similar use case w/ FPGA. We use system DT, but then we convert that into overlays. Bit scary to take away stuff w/ overlays - not very stable. You still have that info in this domain node & then you prepare an overlay & we tell the firmware who owns what. We don't change the regular device, we just change how the DT looks w/ overlays.
    • Benjamin: For us, we put the info in the device node.
    • Tomas: You could still use that & then ___, then you could use Lopper. That becomes an ST-special way at runtime. We could still specify it as we have on the screen & could be translated to what ST needs.
    • Stefano: Is it Linux itself that makes the change, or do you have separate little CPU to do the system-wide config?
    • Benjamin: Initially it is configured secure & then switch from secure to non-secure co-processor. There is still HW control of if this operation is allowed or not.
    • Stefano:
      • Will think through if can provide what you need in execution domains for runtime. This example more focused on boot time.
      • Would this be enough to cover the boot time part of the problem? Assuming you had your own plug-in for Lopper to turn it into what you need.
      • Do we have enough capability in bus firewall itself to protect everything we need to protect?
      • There are only so many ranges you can protect & typically you have more ranges you want to protect than what you can protect. There's no concept of priority or ranking in the domain.
      • Tomas & I were thinking of adding priority attribute to convey priority of protecting a domain.
      • Do we need even more granular priority attribute - at range level instead of execution domain level?
        • e.g. Ethernet controller in access list is more important to protect than MMC controller
        • Then Lopper back-end could know how many slots are available in bus firewall
    • Tomas: Is this something that should be specified, or is this vendor-dependent?
    • Stefano: Most important to protect resources of 1 domain from another domain.
      • e.g. everything in Microblaze domain is important to protect, but which resource in this domain is most important to protect if we don't have enough slots?
    • Tomas: Mental model: If you have a domain & you want to protect everything you can address in this domain, and it’s highest priority, it means all the other domains have to be prevented from touching my domain. We think this is how a user will want to think about it. But then need to consider if we run out of protection registers.
    • Benjamin: So something with lower priority is not protected?
    • Tomas: So then Lopper should spit out warning that only up to priority N was protected
    • Loic: If we want to protect 1 IP, it should be applied. We need 2 options at lopper level: Device accessible when protected, or protected from everybody. Imagine that you want to protect 1 device but we can't b/c lower priority. E.g. Won't pass POS certification. Lopper should always generate same output & black/white.
    • Tomas: If it's only in 1 domain, it should not be accessible if you have enough registers. If it's shared, it needs to be in multiple places. Then, what if not enough registers - in access list, want to prioritize which gets protected. E.g. if have POS registers, must be protected. Then can have other things that are nice to have protected.
    • Mark: So to protect memory from other domains, you have to configure other domains so they can't DMA into it.
    • Stefano: It might be possible to do what you describe, but this is not what this is for, in theory. It's not for configuring SMMU. 1 configuration SoC-wide. Usually fixed # (e.g. 8-12) of ranges that can be specified.
    • Tomas: that's more logical way for user to specify & then have tooling to tell the firmware.
    • Mark: SMMU is fine b/c page table based. Things like RDC aren't. Could get complicated if you have multiple priorities. You need to know in a safety system that everything is protected.
    • Tomas: Tooling needs to report what did/didn't get protected. If do other way, where each thing is specified what needs to be avoided, it's very error-prone.
    • Stefano: We don't mean best effort. We convey the info in the DT. If you say everything with priority 9 must be protected & if you can't do that, you get error in Lopper.
    • Tomas; Not something you discover at runtime that it can't be protected. You have a tool that does the allocation.
    • Mark: Usually it's all or nothing - either you protect everything or it's not enough.
  • Tomas: Lopper
    • Kalray is testing out Lopper
    • Bruce is working on putting it on a more official GitHub
    • Doesn't have the firewall stuff in it yet
    • Transforming System DT -> DT
    • Will send out more info this week

2020-Feb-13

Attended:

  • Loic Pallardy (ST), Mark Dapoz, Rob Herring, Tomas, Stefano Stabellini, Dan Milea, Ed Mooring, Ilias Apalodimas, Joakim Bech, Nathalie Chan King Choy, Pierre (Kalray), Dan Driscoll, Grant Likely (Arm), Dan Milea, krzysztofkepa

Action items

  • Stefano: review memory & reserved memory to make sure it's consistent
  • Tomas/Stefano: We should add FAQ to the System DT write-up
  • Tomas: Ask Bruce to send out Lopper more widely, even if draft, soon
  • Tomas: Drive documenting Xilinx internal discussion on firewalls & share
  • All: Please share your ideas & then we can discuss a proposal
  • Nathalie: Put on next call agenda to continue to discuss configuration lifecycle next meeting

Notes

  • Stefano & Tomas wrote a doc to explain what System DT is about concisely & the use cases & sent to list
    • How does concept of default view benefit the use cases or not? What is the relation between them?
    • What is missing & can anything be explained better?
    • It's a great way to start b/c lots of scattered concepts in System DT. This is 1st attempt to explain how they come together
  • Rob: Have not had a chance to go through it
  • Stefano: What do you think should be covered in the doc @ high level?
    • We have use cases
    • We have interrupts description
    • We have execution domain explanation
    • How chosen & config properties
    • How to describe multiple heterogeneous CPU clusters
    • Memory & Reserved memory
      • Stefano: review memory & reserved memory to make sure it's consistent
    • Tomas/Stefano: We should add FAQ to the System DT write-up
      • How do we do XYZ?
      • How do you add a new device?
      • How do you add protection?
      • How do you deal w/ interrupt that's only accessible from one?
    • Stefano: We do get some good Qs that come up & good to address them
  • Tomas: Hasn't been clear that System DT can be used in multiple ways
    • Host-only tool & shouldn't have to change anything w/ client & just split it up. Linux doesn't have to worry about it.
    • Grant's point that maybe it should be readable directly from Linux -> Default, which has some other interesting properties & makes it easier to put into description of Device Trees
    • Need to look at these 2 cases separately
      • Let's start with host-only b/c less impact
      • Make sure it's backward compatible
    • What do others think?
    • Dan Driscoll: Makes sense to me. Then System DT is available & has some tooling around it & Linux uses what's generated from the System DT?
    • Tomas: Still want to make sure Linux won't choke on a System DT
    • Mark Dapoz: Combined system one is useful for Hypervisor case. Don't have to be backwards compatible with Linux in that case
    • Dan D: Yes, we've done @ Mentor, but it was proprietary. World prefer solution created outside.
  • Mark Dapoz: WR same way & prune it down. In the tooling good if we can do it. Haven't seen any good UI tooling to do this. Very hard to express in text. Need a graphical visualization.
    • Rob: Are any of these tools public?
    • Mark: Don't think any tools exist
    • Tomas: Bruce working on Lopper & sent out to some ppl. Intent is to make it available
      • Tomas: Ask Bruce to send out Lopper more widely, even if draft, soon
      • Initially, need to work out the text format before we get to the graphical tooling
    • Stefano: Have way to assign devices thru DT in Xen. Too difficult for users.
    • Grant had to drop off approx here due to conflict
    • Dan D:
      • Agree we need textual definition nailed down B4 tools. We had evolving content internally & the tools had hard time to keep up
      • Graphical side is very hard. Partitioning is hard without impact to guests. Hard to solve even with tool. SW change often needed to support the tools.
    • Tomas:
      • Should make sure even if we have Python-like tool that we have a C/C++ version that we can read with libfdt & can do runtime tooling as well
      • Lopper is Python on top of dtc & libfdt
      • Can have different data-driven back-end plug-ins
        • Have DT back-end that can prune for 1 or multiple domains
        • Will have RTOS back-end for #defines that can compile in
        • Will have one for Xilinx firmware
    • Dan D: Will Lopper get open sourced? When?
    • Tomas: Yes. TBD under what project. Flexible. Don't want it to be just a Xilinx thing.
    • Dan D: We agree with path forward with System DT & held off on our tool development in this area. Maybe there are things we can contribute to help progress Lopper.
  • Loic: Question about access for assigning peripherals to a domain. May be difficult for end user.
    • Could be shared: GPU banks, some clocks, certain IP
    • e.g. U-Boot, then kernel
    • What about big firewall definitions -> which processor can access or not?
      • You need a table to configure your firewall for peripheral access & memory firewall
      • You need a DT node indicating how the peripherals are accessible & secure/non-secure side
      • If you want to verify access, you can look at this table & not duplicate info in your domain
    • Tomas: In the domain description, you are specifying what memory you are using (might be shared w/ other domains), what devices you're accessing (some might be unique via indirect bus)
      • List serves 2 purposes & need to differentiate. Started internal discussion & should start external
      • Access list so Lopper can prune tree & throw away what's not needed for 1 domain. Doesn't by itself protect any memory or devices.
      • In our devices, we have HW registers that protect regions & devices from other bus masters
      • Usually you have a more limited set of those protections
        • 1:1 mapping
          • In 1 domain & not shared -> should be protected
          • We don't have enough HW to protect everything from everything else
          • But also maybe you want to allow peek/poke
        • You have a domain that's more robust & maybe you need to safety certify it or need more security
          • Maybe can add attribute that says if it's not explicitly shared, then set up protection/firewall
          • Was at high level what we were thinking, but still discussing
        • At more fine grained level, for things we include in domain, be able to override
          • Allow access even if not explicitly protected
        • Domain specification allocation & not spread out for different devices & memories
          • Would mean config info not all in one place
    • Loic:
      • You need all the info for peripherals in 1 place b/c usually have 1 large firewall. Same from memories.
      • What you describe is only understood by Lopper. If we come back to Grant's requirement for kernel to be able to understand.
    • Tomas: If you send unaltered system DT to Linux, it sees everything unless on an indirect bus. My internal model is that Linux is the master & allocates stuff to the other processors. So, if we add features to Linux in conjunction with remoteproc
    • Loic: Disagree. Linux won't be master for firewall management. Might be Cortex-A with TF-A/OP-TEE. Might have dedicated co-processor in charge of security that applies rules & then boot Linux.
    • Stefano: If you have full system DT to platform config processor. Firmware comes up & configures. Similarly, hypervisor coming up & Setting up protection for each domain. If bus firewall already configured & Linux or other OS boots up, maybe not a good idea to pass it a system DT.
      • Spec POV vs. Use-case POV
    • Rob: You can have both use cases
      • Secure world has done some partitioning & Linux can do further partitioning or it gets what's partitioned
    • Stefano: Example of how that will work on Xilinx
      • Firmware configures the whole platform & give System DT to it & it configures bus firewall, etc.
      • Start cortex-A and Xen will do further partitioning
      • I don't think it's a great idea to tell Xen everything incl. what it can't access b/c outside its realm
      • At runtime DT should describe what each part should care about
      • Last leaf of the tree will only see normal DT by then
    • Rob: So you're arguing there is no host tool? Secure world gets the whole thing & strips out.
    • Dan Driscoll: You can do either way & define the partitioning up front
    • Rob: The one to worry about is doing it at runtime. If you can do it at runtime you can do it at build time
    • Tomas: Firmware doesn't have RAM or resources. Can read DT, but can do on tools side. Or, if you have more dynamic
    • Rob: Host tool might not at runtime, but if it works at runtime then will work at build time
    • Loic: Think runtime tool will be too heavy to implement in TF-A or small FSBL
    • Rob: But you're suggesting firewall binding, which is runtime
    • Loic: For me, what's described in domain w/ access, can't imagine TF-A to create & setup the firewall
    • Tomas: For us, Lopper to create table for firmware
    • Stefano: Advantage of generating table - description can also work for hypervisors & different privilege levels. Could cover simply. If we have a well-written but just bus firewall, then not as flexible. Maybe could write it so could be read by Lopper or other tools.
    • Stefano: Don’t imagine that we will pass full System DT to little firmware with tiny memory
    • Tomas: Want system to be flexible enough if you have powerful system that can be dynamic, or have tool that helps you do it statically for low-resource processors
  • Dan D:
    • Shared devices in AMP configuration without hypervisor. Need co-operative understanding between drivers & OS.
    • This area is problematic to us
    • How to annotate DT node what this means to device driver
    • Rob: Doesn’t the HW have to be designed for sharing, so it's implied
    • Yes & no. Lots of cases where … [lost]
    • Rob: IIC bus with spin lock to arbitrate bus
    • Tomas: Normally, you don't share devices. There should be a way to do it in spec. It's still a contract between devices. Don't think have to specify at this level.
    • Stefano: This falls into difficult corner cases.
      • Case where need to show different description if it is/isn't host node
      • Don't have to start on day 1 with this
    • Mark:
      • They may get the device but someone else may own the interrupt controller
      • Do we need a way of describing that
    • Dan D:
      • It's a complex problem that comes up for us when no hypervisor available
      • Curious: If looking into if there a way to address that?
      • We have to modify the OS to handle as needed & it's painful to do for every use case
    • Mark
      • Anything outside of regular memory gets complicated fast
    • Tomas
      • Could have Lopper spit out a warning if a device is shared
      • If you have that problem, the format itself doesn't preclude sharing
      • Maybe just need more hints to the driver - add-on
    • Dan D
      • Need some mechanism to tell end user they are sharing a device & implications
      • Otherwise end user assumes it will just work when they share a device
    • Tomas
      • Maybe this is something you address later for end user when you have a GUI
  • Tomas: Back to Loic's firewall Q
    • Tomas: Drive documenting Xilinx internal discussion on firewalls & share
    • All: Please share your ideas & then we can discuss a proposal
  • Loic: Configuration lifecycle
    • System DT will generate 1 time (Lopper)
    • So have series of DTB
    • So if don't have right updates (kernel w/ new DTB not aligned with firmware one) then can have incorrect access set up
    • Maybe can have U-Boot or kernel that is checking configurations are aligned
    • DTE have some discussions around that
    • Consistency of global system configuration is important
    • Stefano: Overall intent is to make DT more & more stable
      • We all know problem you described still exists
      • Would like the problem to go away over time as we make more stable
      • Do we need to add specific version to parts of it to ameliorate the situation? Hoping DT stability will be good enough so we don't need to do it
      • Loic: Not DT compatibility issue, but customer not updating it properly on the product b/c of async updates.
    • Agree it's good policy for update OTA, but can imagine something goes wrong: Firmware updated & old Linux images
    • Should it be Bootloader that updates DT at runtime?
    • Since system DT will generate several configurations, have question regarding lifecycle
    • Stefano: Master for original info is the original DT input to Lopper. Then everything else falls from there.
    • Tomas: But good Q on how you keep things in sync w/ different teams. Customers will screw up.
    • Loic; Need a way to mention to customer when they update, impact of using partial vs full image
    • Nathalie: Put on next call agenda to continue to discuss configuration lifecycle next meeting
    • Tomas: We've all had issues with DT coming from all over the place & not generated in nice fashion, even when you have the tools available

2020-Jan-22

Attended:

Joakim Bech, Tomas Evensen, Ilias, Rob Herring, Mathieu Poirier, Stefano Stabellini, Ed Mooring, Mark Dapoz, Nathalie Chan King Choy, Bruce Ashfield, Loic Pallardy (ST), Etsam Anjum (MGC), Dan Driscoll, Saravanna Kannan (Google)

Action items:

  • Stefano: Document a little bit more how the model works
    • Remove reserved memory
    • Add top-level use-case FAQ (e.g. how to do peripheral assignment to SW)
    • Consider putting a qualifier word before "domain" to make it more specific
  • Everyone: Try to poke holes in the model. Good to have hard questions to think through & answer
  • Rob: Prototype proposal of changing root
  • Nathalie: co-ordinate next call over email (2 weeks from now doesn't work b/c Rob can't make it)

Notes:

  • Tomas: Recap:
    • Trying to expand current concept of device tree to describe the whole system
      • Add features to talk about multiple CPU clusters, multiple masters
      • Add config info to say which resources (CPU, memory, I/O device) belong to which domain
    • Look at from physical & virtual point-of-view: Would like to be able to describe both AMP & hypervisor use case.
    • Splitting how the HW looks from the configuration
      • DT is more about hardware description than configuration
      • Different ppl are coming with the different info
        • HW info: Board, SoC, vendor
        • Who decides how you split up the resources (e.g. between Linux, FreeRTOS, Zephyr) is different (e.g. System architect)
    • Stefano: added some features & not yet made proposal for config
  • Stefano: Rob suggested to write out both HW (new CPU clusters) & configuration (e.g. OpenAMP domain): How do they interact w/ existing?
    • Writing an overview was useful to get ppl on the same page & how these features fit together
    • Highlighted at least 1 problem:
      • Most top level nodes are to describe the HW (memory, amba bus, cpus).
      • Originally put OpenAMP domain under chosen to mark as configuration. Not really final destination for these domains, but made easy to identify the config & not drastically change structure of DT.
      • Reserved memory is one of few nodes that's already there for configuration. So, why not use that for describing memory? But, this is a bit awkward w/ multiple domains & multiple OS. You want 1 chosen & 1 reserved memory for each domain.
  • Rob:
    • Think you captured it pretty well
    • Domain is probably an over-used term
      • HW domains
      • Each HW domain is 1 or more config domains (e.g. R5, Cortex-A, may be divided into secure/non-secure)
      • Stefano: Agree: Domain is overloaded term
      • Loic: In ST we call it "execution context", where we can run SW component. Could be Cortex A secure on non-secure, or Cortex M. And, all the peripherals we can access.
      • Tomas: We had tried "execution context" in past & no one had understood it. Xen is calling this "domains". Tricky: In some SoCs this is very configurable. Don't want to change the structure of the DT if you change the configuration.
  • Stefano:
    • Makes sense to keep HW description all together & immutable
      • Used CPU attribute
      • Access list that points to buses
      • Memory was controversial case. Had 2 ways:
        • access list (could go to reserved memory - bit controversial),
        • special memory attribute (memory config carve out for domain)
          • Top-level memory node describes physical memory
          • Config of memory & what devices accessible & what cpu cluster we're running described in the domain under chosen (or clearly config section)
  • Discussion on how much effort should go into making it possible for Linux to understand a System DT. Grant urges to do so for default case. Otherwise, Lopper gives "legacy" OS a DT that it can understand.
    • Rob:
      • Doing that w/ memory is trade-off for how OS parses its memory
        • OS can't look at common node describing all of memory
    • Stefano:
      • Compatibility w/ legacy OS & domain description is bit of a balance
    • Rob:
      • If we change how OS parses, OS has to know how to parse both ways b/c might not have System DT. Legacy OS isn't necessarily "legacy" b/c not getting rid of it. With tool, only have to consider 1 way of doing it.
    • Stefano:
      • Just realized an assumption I had: A "legacy" OS will only have to support default. Either we'll get System DT will give the "legacy" OS the default view or have tool Lopper to give the OS the info it needs.
    • Rob:
      • So you're not going to run Linux on R5?
    • Tomas:
      • We intended that you would run Lopper first if it's the secondary OS.
      • To get system DT to feed directly into OS came later. Figured System DT is on host side & would always run Lopper. Grant wanted to see if we could have Linux understand System DT.
    • Rob:
      • Is it worth the messiness to avoid running Lopper on 1 view of the system vs. doing something more symmetrical & cleaner?
    • Tomas:
      • For us, RTOS won't use DT directly. It will use DT to get a .h with #defines in them. B/c resource constrained.
    • Rob:
      • Think that's true today but will diminish over time. Think more processors will run Linux in same system
    • Tomas:
      • You're saying when running multiple Linux instances, 1 will get default & others will need Lopper
    • Etsam:
      • Think system Dt should just work if only 1 execution context in system (just 1 Linux), or if there are multiple contexts then have to run Lopper b/c have to do memory partitioning & device assignment.
      • Tomas; Think this is Stefano's intention
      • Stefano: Yes
    • Stefano:
      • Rob suggesting: Is it worth trying to maintain max compatibility if makes things awkward?
      • We could maintain backward compatibility, but it doesn't have to be super nice. Will make it easier to write changes to spec if more seamless & not revolutionary. Then if want to run "legacy" OS, it will boot for primary case.
      • Trying to make multiple domain case w/ Lopper nice. Reserved memory makes this awkward
    • Etsam:
      • For multiple execution contexts or multiple domains, currently we are putting all the resource assignment info into chosen node. Problem is that OS will have to parse that info.
      • Dan proposed if we separate out the resource assignment to domains vs. what's needed to capture heterogeneity in SoCs
      • Even with that, having multiple execution contexts will need a tool to be run
      • If we move the domain concept out of the main system DT, the changes in OS will be minimal. Will not need to parse domain nodes in chosen
    • Stefano:
      • Have not had a chance to read Dan's yet, but think my proposal will require minimal changes. No change for legacy OS in default. Lopper use -> don't need to change OS.
      • If want to enhance an OS to understand system DT w/o Lopper at runtime, but amt of new info to read is quite minimal, if it knows which CPU cluster to use, all the info is there. There is a bit of link jumping & reference checking, but just 3 new attributes to read. Think it will be feasible to add support for that
      • And, is it worth optimizing for this case when you can use Lopper & not need any OS changes?
    • Rob: Back to my proposal of domains & changing what the root is
      • If we look at flattened DT, that may only require zeroing out the top level node to make domain at 0 node the root node, which would be a very minimal change
      • Or, at most, when you unflatten it, you could point to a different location & make that your root
      • If you have everything under a domain node & nothing else outside, then it would be a trivial change to the OS. Or, even the bootloader could do it before you give it to the OS. You could do Lopper as part of your boot flow.
    • Etsam:
      • Which OS will pick which node from the domain nodes when you have multiple execution contexts?
    • Rob:
      • You could say that the default node is domain 0. At some point, you have to pick which is which. Probably a couple ways to do that.
    • Tomas:
      • If you do that, you have decided on the domains ahead, but the domains are decided later by system architect (e.g. when you have TrustZone) with same device & multiple domains
    • Rob:
      • Probably need 2 levels of domains:
        • HW domains: Pretty much fixed: Cortex-A, R5, microcontroller
        • Within those, you have config domains which handle asymmetric cases, or TZ, or hyper
    • Dan:
      • Could have hypervisor running 4 different guests. A cluster could be multiple domains from a SW perspective. So, can't break up HW into subsystem cluster. It's how you map the SW to any execution context.
    • Rob:
      • You'd have 2 levels: 1 level is dictated by HW. Within a HW domain, you're creating config or openamp domains
    • Dan:
      • OK, in the end, yoy have a HW cluster that all the devices & buses fall under. Right now there's no relationship back to which CPU cluster
    • Stefano:
      • What about device that has been assigned to one of the OpenAMP domains. How would you express that?
    • Rob:
      • Good question. If the devices are all viewable by all the HW domains, then it would be at the top level.
    • Tomas:
      • Way to think about our system is CPU, memory, devices are fairly independent. Can assign as you want, mostly.
      • Interrupt controllers are some exceptions, but Stefano has way of expressing that.
      • General use case is to have someone else select from HW resources & create the domains. For CPU clusters, how do you tie those to the fixed HW - think Stefano was doing w/ these different buses.
    • Stefano:
      • If you consider that devices remain top level (e.g. amba), and link to them under Rob's domains. Not too different. Then OpenAMP domain would be directly under cpu clusters.
    • Loic:
      • Benjamin from ST has proposal for domain controller. Working on for kernel & u-boot. He proposed some bindings in each peripheral node to have info about which execution context can access. It's directly defined in the peripheral's node, like pin control. You can then say which domain it is associated.
      • Tomas: Then do you have to change tree when you reconfigure?
      • Dan: Would have to show that it could refer back to multiple execution contexts, whether it can access it or not. Not to do with domain definition, just the HW association
      • Loic: Yes
      • Dan: Seem have same issue w/ interrupt controller
      • Stefano: Solved in last example by describing all interrupt controllers. MPSoC has multiple interrupt controllers.
      • Dan: Is it changing the bindings or adding new?
      • Stefano: Thanks to Grant's suggestion, could do without any change. You use as interrupt parent of all devices like an interrupt muxer.
      • Dan: Then Lopper has to generate the right piece that this is the interrupt controller that I have
      • Rob: We only solved how to describe more than 1 parent, but not how to select which parent is the correct one. If you have HW domain, then you can stick it in HW domain.
      • Stefano: There is also HW restrictions where something is not visible or accessible between clusters. This is currently done with address map & indirect bus.
      • Dan: OK, in that case the info is all in the system DT & Lopper not necessary. So, we are discussing some cases where don’t need Lopper & other cases where Lopper makes life easier. At some point, we have to choose.
      • Tomas: We didn't originally think it could be backward compatible until we had the conversation w/ Grant. In a way, having default domain makes a lot of sense.
  • Tomas: Are there unsolved issues being able to express all the domains (SW context), HW that is only attached to only 1 cpu cluster & not others? Or, are all the issues kind of solved in your mind & more syntax to work out?
    • Stefano:
      • Not aware of open issues for the heterogeneous clusters case. It's about syntax & how we want to express & make nicer for 1 type of parsing
      • There are issues for VM use case. Want to wait for these ideas to settle 1st. Think could be solved additions
        • Memory for VM needs physical & pseudo-physical ranges
        • Where to place virtual-only devices
    • Rob:
      • Disagree that there's not open issues & it's just syntax. Syntax is open issues & need to know that they will work for the different cases.
      • Feel like we've gone down this path & each step there's different issues to solve & thinking about if we should back up & go down a different path w/ less issues to solve
    • Tomas:
      • Want to make sure that the model holds up
      • Lots of good energy on the topic. Think we all want the same thing & just need to clarify.
Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️