QEMU NVMe notes - animeshtrivedi/notes GitHub Wiki
Github: https://github.com/qemu/qemu/tree/master/hw/nvme
src is in hw/nvme/
-
ctrl.chas 9K lines of code where majority of the logic is -
ns.cnamespace related concepts -
nvme.himplementation header files -
subsys.cnvme subsystem related code
QEMU NVMe details: https://kvm-forum.qemu.org/2021/jensen-qemu-emulated-nvme.pdf
ctrl.c has static bool nvme_check_params(NvmeCtrl *n, Error **errp) for param parsing and sanity checks for controllers
atr@f20u24:~$ sudo /home/atr/local/sbin//nvme show-regs /dev/nvme0 -H
cap : 0x4018200f0107ff
NVM Subsystem Shutdown Enhancements Supported (NSSES): Not Supported
Controller Ready With Media Support (CRWMS): Not Supported
Controller Ready Independent of Media Support (CRIMS): Not Supported
NVM Subsystem Shutdown Supported (NSSS): Not Supported
Controller Memory Buffer Supported (CMBS): The Controller Memory Buffer is Not Supported
Persistent Memory Region Supported (PMRS): The Persistent Memory Region is Not Supported
Memory Page Size Maximum (MPSMAX): 65536 bytes
Memory Page Size Minimum (MPSMIN): 4096 bytes
Controller Power Scope (CPS): Not Reported
Boot Partition Support (BPS): No
Command Sets Supported (CSS): NVM command set is Supported
One or more I/O Command Sets are Supported
Only Admin Command Set Supported
NVM Subsystem Reset Supported (NSSRS): No
Doorbell Stride (DSTRD): 4 bytes
Timeout (TO): 7500 ms
Arbitration Mechanism Supported (AMS): Weighted Round Robin with Urgent Priority Class is Not supported
Contiguous Queues Required (CQR): Yes
Maximum Queue Entries Supported (MQES): 2048
version : 0x10400
NVMe specification 1.4.0
intms : 0
Interrupt Vector Mask Set (IVMS): 0
intmc : 0
Interrupt Vector Mask Clear (IVMC): 0
cc : 0x460061
Controller Ready Independent of Media Enable (CRIME): Disabled
I/O Completion Queue Entry Size (IOCQES): 16 bytes
I/O Submission Queue Entry Size (IOSQES): 64 bytes
Shutdown Notification (SHN): No notification; no effect
Arbitration Mechanism Selected (AMS): Round Robin
Memory Page Size (MPS): 4096 bytes
I/O Command Set Selected (CSS): All supported I/O Command Sets
Enable (EN): Yes
csts : 0x1
Shutdown Type (ST): Controller
Processing Paused (PP): No
NVM Subsystem Reset Occurred (NSSRO): No
Shutdown Status (SHST): Normal operation (no shutdown has been requested)
Controller Fatal Status (CFS): False
Ready (RDY): Yes
nssr : 0
NVM Subsystem Reset Control (NSSRC): 0
aqa : 0x1f001f
Admin Completion Queue Size (ACQS): 32
Admin Submission Queue Size (ASQS): 32
asq : 0x10ee3d000
Admin Submission Queue Base (ASQB): 10ee3d
acq : 0x110a36000
Admin Completion Queue Base (ACQB): 110a36
cmbloc : 0
Controller Memory Buffer feature is not supported
cmbsz : 0
Controller Memory Buffer feature is not supported
bpinfo : 0
Active Boot Partition ID (ABPID): 0
Boot Read Status (BRS): No Boot Partition read operation requested
Boot Partition Size (BPSZ): 0
bprsel : 0
Boot Partition Identifier (BPID): 0
Boot Partition Read Offset (BPROF): 0
Boot Partition Read Size (BPRSZ): 0
bpmbl : 0
Boot Partition Memory Buffer Base Address (BMBBA): 0
cmbmsc : 0
Controller Base Address (CBA): 0
Controller Memory Space Enable (CMSE): 0
Capabilities Registers Enabled (CRE): CMBLOC and CMBSZ registers are NOT enabled
cmbsts : 0
Controller Base Address Invalid (CBAI): 0
cmbebs : 0
CMB Elasticity Buffer Size Base (CMBWBZ): 0
Read Bypass Behavior : memory reads not conflicting with memory writes in the CMB Elasticity Buffer MAY bypass those memory writes
CMB Elasticity Buffer Size Units (CMBSZU): Bytes
cmbswtp : 0
CMB Sustained Write Throughput (CMBSWTV): 0
CMB Sustained Write Throughput Units (CMBSWTU): Bytes/second
nssd : 0
NVM Subsystem Shutdown Control (NSSC): 0
crto : 0
CRIMT : 0 secs
CRWMT : 0 secs
pmrcap : 0
Controller Memory Space Supported (CMSS): Referencing PMR with host supplied addresses is Not Supported
Persistent Memory Region Timeout (PMRTO): 0
Persistent Memory Region Write Barrier Mechanisms (PMRWBM): 0
Persistent Memory Region Time Units (PMRTU): PMR time unit is 500 milliseconds
Base Indicator Register (BIR): 0
Write Data Support (WDS): Write data to the PMR is not supported
Read Data Support (RDS): Read data from the PMR is not supported
pmrctl : 0
Enable (EN): PMR is Disabled
pmrsts : 0
Controller Base Address Invalid (CBAI): 0
Health Status (HSTS): Normal Operation
Not Ready (NRDY): The Persistent Memory Region is Not Ready to process PCI Express memory read and write requests
Error (ERR): 0
pmrebs : 0
PMR Elasticity Buffer Size Base (PMRWBZ): 0
Read Bypass Behavior : memory reads not conflicting with memory writes in the PMR Elasticity Buffer MAY bypass those memory writes
PMR Elasticity Buffer Size Units (PMRSZU): Bytes
pmrswtp : 0
PMR Sustained Write Throughput (PMRSWTV): 0
PMR Sustained Write Throughput Units (PMRSWTU): Bytes/second
pmrmscl : 0
Controller Base Address (CBA): 0
Controller Memory Space Enable (CMSE): 0
pmrmscu : 0
Controller Base Address (CBA): 0
atr@u24clean:~$ sudo nvme id-ctrl /dev/nvme0 -H
NVME Identify Controller:
vid : 0x1b36
ssvid : 0x1af4
sn : nvme-dev
mn : QEMU NVMe Ctrl
fr : 9.0.2
rab : 6
ieee : 525400
cmic : 0
[3:3] : 0 ANA not supported
[2:2] : 0 PCI
[1:1] : 0 Single Controller
[0:0] : 0 Single Port
mdts : 7
cntlid : 0
ver : 0x10400
rtd3r : 0
rtd3e : 0
oaes : 0x100
[31:31] : 0 Discovery Log Change Notice Not Supported
[27:27] : 0 Zone Descriptor Changed Notices Not Supported
[15:15] : 0 Normal NSS Shutdown Event Not Supported
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
[9:9] : 0 Firmware Activation Notices Not Supported
[8:8] : 0x1 Namespace Attribute Changed Event Supported
ctratt : 0x8000
[19:19] : 0 Flexible Data Placement Not Supported
[15:15] : 0x1 Extended LBA Formats Supported
[14:14] : 0 Delete NVM Set Not Supported
[13:13] : 0 Delete Endurance Group Not Supported
[12:12] : 0 Variable Capacity Management Not Supported
[11:11] : 0 Fixed Capacity Management Not Supported
[10:10] : 0 Multi Domain Subsystem Not Supported
[9:9] : 0 UUID List Not Supported
[8:8] : 0 SQ Associations Not Supported
[7:7] : 0 Namespace Granularity Not Supported
[6:6] : 0 Traffic Based Keep Alive Not Supported
[5:5] : 0 Predictable Latency Mode Not Supported
[4:4] : 0 Endurance Groups Not Supported
[3:3] : 0 Read Recovery Levels Not Supported
[2:2] : 0 NVM Sets Not Supported
[1:1] : 0 Non-Operational Power State Permissive Not Supported
[0:0] : 0 128-bit Host Identifier Not Supported
rrls : 0
cntrltype : 1
[7:2] : 0 Reserved
[1:0] : 0x1 I/O Controller
fguid : 00000000-0000-0000-0000-000000000000
crdt1 : 0
crdt2 : 0
crdt3 : 0
nvmsr : 0
[1:1] : 0 NVM subsystem Not part of an Enclosure
[0:0] : 0 NVM subsystem Not part of a Storage Device
vwci : 0
[7:7] : 0 VPD Write Cycles Remaining field is Not valid.
[6:0] : 0 VPD Write Cycles Remaining
mec : 0
[1:1] : 0 NVM subsystem Not contains a Management Endpoint on a PCIe port
[0:0] : 0 NVM subsystem Not contains a Management Endpoint on an SMBus/I2C port
oacs : 0x12a
[10:10] : 0 Lockdown Command and Feature Not Supported
[9:9] : 0 Get LBA Status Capability Not Supported
[8:8] : 0x1 Doorbell Buffer Config Supported
[7:7] : 0 Virtualization Management Not Supported
[6:6] : 0 NVMe-MI Send and Receive Not Supported
[5:5] : 0x1 Directives Supported
[4:4] : 0 Device Self-test Not Supported
[3:3] : 0x1 NS Management and Attachment Supported
[2:2] : 0 FW Commit and Download Not Supported
[1:1] : 0x1 Format NVM Supported
[0:0] : 0 Security Send and Receive Not Supported
acl : 3
aerl : 3
frmw : 0x3
[5:5] : 0 Multiple FW or Boot Update Detection Not Supported
[4:4] : 0 Firmware Activate Without Reset Not Supported
[3:1] : 0x1 Number of Firmware Slots
[0:0] : 0x1 Firmware Slot 1 Read-Only
lpa : 0x7
[6:6] : 0 Telemetry Log Data Area 4 Not Supported
[5:5] : 0 LID 0x0, Scope of each command in LID 0x5, 0x12, 0x13 Not Supported
[4:4] : 0 Persistent Event log Not Supported
[3:3] : 0 Telemetry host/controller initiated log page Not Supported
[2:2] : 0x1 Extended data for Get Log Page Supported
[1:1] : 0x1 Command Effects Log Page Supported
[0:0] : 0x1 SMART/Health Log Page per NS Supported
elpe : 0
[7:0] : 0 (0's based) Error Log Page Entries (ELPE)
npss : 0
[7:0] : 0 (0's based) Number of Power States Support (NPSS)
avscc : 0
[0:0] : 0 Admin Vendor Specific Commands uses Vendor Specific Format
apsta : 0
[0:0] : 0 Autonomous Power State Transitions Not Supported
wctemp : 343
[15:0] : 70 °C (343 K) Warning Composite Temperature Threshold (WCTEMP)
cctemp : 373
[15:0] : 100 °C (373 K) Critical Composite Temperature Threshold (CCTEMP)
mtfa : 0
hmpre : 0
hmmin : 0
tnvmcap : 0
[127:0] : 0
Total NVM Capacity (TNVMCAP)
unvmcap : 0
[127:0] : 0
Unallocated NVM Capacity (UNVMCAP)
rpmbs : 0
[31:24]: 0 Access Size
[23:16]: 0 Total Size
[5:3] : 0 Authentication Method
[2:0] : 0 Number of RPMB Units
edstt : 0
dsto : 0
fwug : 0
kas : 0
hctma : 0
[0:0] : 0 Host Controlled Thermal Management Not Supported
mntmt : 0
[15:0] : -273 °C (0 K) Minimum Thermal Management Temperature (MNTMT)
mxtmt : 0
[15:0] : -273 °C (0 K) Maximum Thermal Management Temperature (MXTMT)
sanicap : 0
[31:30] : 0 Additional media modification after sanitize operation completes successfully is not defined
[29:29] : 0 No-Deallocate After Sanitize bit in Sanitize command Supported
[2:2] : 0 Overwrite Sanitize Operation Not Supported
[1:1] : 0 Block Erase Sanitize Operation Not Supported
[0:0] : 0 Crypto Erase Sanitize Operation Not Supported
hmminds : 0
hmmaxd : 0
nsetidmax : 0
endgidmax : 0
anatt : 0
anacap : 0
[7:7] : 0 Non-zero group ID Not Supported
[6:6] : 0 Group ID does change
[4:4] : 0 ANA Change state Not Supported
[3:3] : 0 ANA Persistent Loss state Not Supported
[2:2] : 0 ANA Inaccessible state Not Supported
[1:1] : 0 ANA Non-optimized state Not Supported
[0:0] : 0 ANA Optimized state Not Supported
anagrpmax : 0
nanagrpid : 0
pels : 0
domainid : 0
megcap : 0
sqes : 0x66
[7:4] : 0x6 Max SQ Entry Size (64)
[3:0] : 0x6 Min SQ Entry Size (64)
cqes : 0x44
[7:4] : 0x4 Max CQ Entry Size (16)
[3:0] : 0x4 Min CQ Entry Size (16)
maxcmd : 0
nn : 256
oncs : 0x15d
[10:10] : 0 All Fast Copy Not Supported
[9:9] : 0 Copy Single Atomicity Not Supported
[8:8] : 0x1 Copy Supported
[7:7] : 0 Verify Not Supported
[6:6] : 0x1 Timestamp Supported
[5:5] : 0 Reservations Not Supported
[4:4] : 0x1 Save and Select Supported
[3:3] : 0x1 Write Zeroes Supported
[2:2] : 0x1 Data Set Management Supported
[1:1] : 0 Write Uncorrectable Not Supported
[0:0] : 0x1 Compare Supported
fuses : 0
[0:0] : 0 Fused Compare and Write Not Supported
fna : 0
[3:3] : 0 Format NVM Broadcast NSID (FFFFFFFFh) Supported
[2:2] : 0 Crypto Erase Not Supported as part of Secure Erase
[1:1] : 0 Crypto Erase Applies to Single Namespace(s)
[0:0] : 0 Format Applies to Single Namespace(s)
vwc : 0x7
[2:1] : 0x3 The Flush command supports NSID set to FFFFFFFFh
[0:0] : 0x1 Volatile Write Cache Present
awun : 0
awupf : 0
icsvscc : 0
[0:0] : 0 NVM Vendor Specific Commands uses Vendor Specific Format
nwpc : 0
[2:2] : 0 Permanent Write Protect Not Supported
[1:1] : 0 Write Protect Until Power Supply Not Supported
[0:0] : 0 No Write Protect and Write Protect Namespace Not Supported
acwu : 0
ocfs : 0x3
[3:3] : 0 Controller Copy Format 3h Not Supported
[2:2] : 0 Controller Copy Format 2h Not Supported
[1:1] : 0x1 Controller Copy Format 1h Supported
[0:0] : 0x1 Controller Copy Format 0h Supported
sgls : 0x1
[21:21]: 0 Transport SGL Data Block Descriptor Not Supported
[20:20]: 0 Address Offsets Not Supported
[19:19]: 0 Metadata Pointer Containing SGL Descriptor is Not Supported
[18:18]: 0 SGL Length Larger than Buffer Not Supported
[17:17]: 0 Byte-Aligned Contig. MD Buffer Not Supported
[16:16]: 0 SGL Bit-Bucket Not Supported
[15:8] : 0 SGL Descriptor Threshold
[2:2] : 0 Keyed SGL Data Block descriptor Not Supported
[1:0] : 0x1 Scatter-Gather Lists Supported. No Dword alignment required.
mnan : 0
maxdna : 0
maxcna : 0
oaqd : 0
subnqn : nqn.2019-08.org.qemu:nvme-dev
ioccsz : 0
iorcsz : 0
icdoff : 0
fcatt : 0
[0:0] : 0 Dynamic Controller Model
msdbd : 0
ofcs : 0
[0:0] : 0 Disconnect command Not Supported
ps 0 : mp:25.00W operational enlat:16 exlat:4 rrt:0 rrl:0
rwt:0 rwl:0 idle_power:- active_power:-
active_power_workload:-
There is worker process per SQ: ctrl.c https://github.com/qemu/qemu/blob/master/hw/nvme/ctrl.c#L7345
static void nvme_process_sq(void *opaque)
{
NvmeSQueue *sq = opaque;
NvmeCtrl *n = sq->ctrl;
NvmeCQueue *cq = n->cq[sq->cqid];
...
}From this function: https://github.com/qemu/qemu/blob/master/hw/nvme/ctrl.c#L7404C3-L7405C3
status = sq->sqid ? nvme_io_cmd(n, req) :
nvme_admin_cmd(n, req);https://github.com/qemu/qemu/blob/master/hw/nvme/ctrl.c#L478C1-L514C1
static bool nvme_update_ruh(NvmeCtrl *n, NvmeNamespace *ns, uint16_t pid)
{
NvmeEnduranceGroup *endgrp = ns->endgrp;
NvmeRuHandle *ruh;
NvmeReclaimUnit *ru;
NvmeFdpEvent *e = NULL;
uint16_t ph, rg, ruhid;
if (!nvme_parse_pid(ns, pid, &ph, &rg)) {
return false;
}
ruhid = ns->fdp.phs[ph];
ruh = &endgrp->fdp.ruhs[ruhid];
ru = &ruh->rus[rg];
if (ru->ruamw) {
if (log_event(ruh, FDP_EVT_RU_NOT_FULLY_WRITTEN)) {
e = nvme_fdp_alloc_event(n, &endgrp->fdp.host_events);
e->type = FDP_EVT_RU_NOT_FULLY_WRITTEN;
e->flags = FDPEF_PIV | FDPEF_NSIDV | FDPEF_LV;
e->pid = cpu_to_le16(pid);
e->nsid = cpu_to_le32(ns->params.nsid);
e->rgid = cpu_to_le16(rg);
e->ruhid = cpu_to_le16(ruhid);
}
/* log (eventual) GC overhead of prematurely swapping the RU */
nvme_fdp_stat_inc(&endgrp->fdp.mbmw, nvme_l2b(ns, ru->ruamw));
}
ru->ruamw = ruh->ruamw;
return true;
}media bytes written to reflect this nvme_fdp_stat_inc(&endgrp->fdp.mbmw, nvme_l2b(ns, ru->ruamw)); that the old RUH will be recycled. The amount of valid data written is not taken into the account. The real device may do it better.
https://github.com/qemu/qemu/blob/master/hw/nvme/nvme.h#L163
Follows the NVMe 2.1 specification figure 289
static const uint8_t nvme_fdp_evf_shifts[FDP_EVT_MAX] = {
/* Host events */
[FDP_EVT_RU_NOT_FULLY_WRITTEN] = 0,
[FDP_EVT_RU_ATL_EXCEEDED] = 1,
[FDP_EVT_CTRL_RESET_RUH] = 2,
[FDP_EVT_INVALID_PID] = 3,
/* CTRL events */
[FDP_EVT_MEDIA_REALLOC] = 32,
[FDP_EVT_RUH_IMPLICIT_RU_CHANGE] = 33,
};set-feature where the ruh->event_filter is updated.
static uint16_t nvme_set_feature_fdp_events(NvmeCtrl *n, NvmeNamespace *ns,
NvmeRequest *req)
{
NvmeCmd *cmd = &req->cmd;
uint32_t cdw11 = le32_to_cpu(cmd->cdw11);
uint16_t ph = cdw11 & 0xffff;
uint8_t noet = (cdw11 >> 16) & 0xff;
uint16_t ret, ruhid;
uint8_t enable = le32_to_cpu(cmd->cdw12) & 0x1;
uint8_t event_mask = 0;
unsigned int i;
g_autofree uint8_t *events = g_malloc0(noet);
NvmeRuHandle *ruh = NULL;
assert(ns);
if (!n->subsys || !n->subsys->endgrp.fdp.enabled) {
return NVME_FDP_DISABLED | NVME_DNR;
}
if (!nvme_ph_valid(ns, ph)) {
return NVME_INVALID_FIELD | NVME_DNR;
}
ruhid = ns->fdp.phs[ph];
ruh = &n->subsys->endgrp.fdp.ruhs[ruhid];
ret = nvme_h2c(n, events, noet, req);
if (ret) {
return ret;
}
for (i = 0; i < noet; i++) {
event_mask |= (1 << nvme_fdp_evf_shifts[events[i]]);
}
if (enable) {
ruh->event_filter |= event_mask;
} else {
ruh->event_filter = ruh->event_filter & ~event_mask;
}
return NVME_SUCCESS;
}https://github.com/qemu/qemu/blob/master/hw/nvme/ctrl.c#L4579
static uint16_t nvme_io_mgmt_send(NvmeCtrl *n, NvmeRequest *req)
{
NvmeCmd *cmd = &req->cmd;
uint32_t cdw10 = le32_to_cpu(cmd->cdw10);
uint8_t mo = (cdw10 & 0xff);
switch (mo) {
case NVME_IOMS_MO_NOP:
return 0;
case NVME_IOMS_MO_RUH_UPDATE:
return nvme_io_mgmt_send_ruh_update(n, req);
default:
return NVME_INVALID_FIELD | NVME_DNR;
};
}nvme_io_mgmt_send_ruh_update:
https://github.com/qemu/qemu/blob/master/hw/nvme/ctrl.c#L4595
static uint16_t nvme_io_cmd(NvmeCtrl *n, NvmeRequest *req)
{
...
}ctrl.c
static void nvme_do_write_fdp(NvmeCtrl *n, NvmeRequest *req, uint64_t slba,
uint32_t nlb)
{
if (dtype != NVME_DIRECTIVE_DATA_PLACEMENT ||
!nvme_parse_pid(ns, pid, &ph, &rg)) {
ph = 0;
rg = 0;
}
printf(" pid is %d and ph is %d and rg %d \n", pid, ph, rg);
}- So I pass anything which is not 0x2 then it will default to (0,0) which made sense to what was happening before
- prints what I expects it to print with the split parsing ;)
I am guess subsystem serial is initialized here, subsys->serial 111
VNC server running on 127.0.0.1:5900
atr> pid is 16386 and ph is 2 (which maps to ruhid: 6) and rg 2
atr> nlb is 8 and ru->ruamw is 196608
atr> pid is 16386 and ph is 2 (which maps to ruhid: 6) and rg 2
atr> nlb is 8 and ru->ruamw is 196600
atr> pid is 255 and ph is 0 (which maps to ruhid: 4) and rg 0
atr> nlb is 8 and ru->ruamw is 196608
There seems to be some bug. The (0,0) gets updated and reflected immediately. Even though the value is updated. So I suspect may be it is then the reading the logic.
./ns.c:423: ruh->ruamw = endgrp->fdp.runs >> ns->lbaf.ds;
./ns.c:426: ruh->rus[rg].ruamw = ruh->ruamw;
./ns.c:509: ruh->ruamw = endgrp->fdp.runs >> ns->lbaf.ds;
./ns.c:512: ruh->rus[rg].ruamw = ruh->ruamw;
Must be between these lines (grep "ruamw"). Other accesses are from the write path, that I just checked. OK, it was a bug in my code, where total_size variable was re-used between the calls.
https://github.com/qemu/qemu/blob/master/hw/nvme/subsys.c#L210
static bool nvme_subsys_setup(NvmeSubsystem *subsys, Error **errp)
{
const char *nqn = subsys->params.nqn ?
subsys->params.nqn : subsys->parent_obj.id;
///
}What is the role of serial here with NVMe controller anyways? https://github.com/qemu/qemu/blob/master/hw/nvme/subsys.c#L86
if (!subsys->serial) {
subsys->serial = g_strdup(n->params.serial);
} else if (strcmp(subsys->serial, n->params.serial)) {
error_setg(errp, "invalid controller serial");
return -1;
}The first controller sets the serial property, and then the second one must match it. What is the role of the serial? it is not used or checked anywhere
./nvme.h:104: char *serial;
./nvme.h:520: char *serial;
./ctrl.c:30: * -device nvme,serial=<serial>,id=<bus_name>, \
./ctrl.c:7803: if (!params->serial) {
./ctrl.c:7804: error_setg(errp, "serial property not set");
./ctrl.c:8182: "nqn.2019-08.org.qemu:%s", n->params.serial); <-- this generate subsys ID
./ctrl.c:8200: strpadcpy((char *)id->sn, sizeof(id->sn), n->params.serial, ' ');
./ctrl.c:8348: * Set PF's serial value to a new string memory to prevent 'serial'
./ctrl.c:8351: n->params.serial = g_strdup(pn->params.serial);
./ctrl.c:8428: DEFINE_PROP_STRING("serial", NvmeCtrl, params.serial),
./subsys.c:84: if (!subsys->serial) {
./subsys.c:85: subsys->serial = g_strdup(n->params.serial);
./subsys.c:86: } else if (strcmp(subsys->serial, n->params.serial)) {
./subsys.c:87: printf("invalid controller serial subsys->serial %s n->params.serial %s \n", subsys->serial, n->params.serial);
./subsys.c:88: error_setg(errp, "invalid controller serial");