Logbook - input-output-hk/mithril GitHub Wiki

This page records a log of the team’s work, thinking, discussions, pains, joys, events, and experiences that happen on a daily basis. It’s supposed to be a kind of Stream of consciousness that can later be searched, analyzed, reviewed in more formal retrospectives.

Rules:

Log days in reverse chronological order, recent first
When the stream is getting too big, such that github refuses to render do the following:
- extract biggest logical section (half-year or quarter) out of this into a correspondingly named Logbook-Year-Hn.md or Logbook-Year-Qn.md page
- update links on top of this page to make it easier to navigate try to fix all links in the wiki at least
- notify the team
Move (and link) recurring sections to relevant dedicated pages.

History

July 24

2024-H1

2023-H2

2023-H1

2022-H2

2022-H1

2021

2021

July 2024

2024-07-26

Today, the team has worked on these topics:
- Release 2430 distribution #1830: the issue has been groomed. We have created the 2430.0-pre pre-release distribution. It has been deployed on the pre-release-preview network on which we have activated the certification of Cardano transaction and deployed Cardano node 9.1.0. We expect to have the first certificate created for this new type of data in a few days once enough SPOs have upgraded their Mithril signer node. The release is expected for early next week. The following PRs have been created, reviewed and merged: Upgrade Rust dependencies #1858, Upgrade release-mainnet aggregator VM data disk #1859 and Update CHANGELOG for 2430 distribution release #1860
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: we have kept pairing on the PR Implement signable and artifact builders for Cardano Stake Distribution #1847 to fix some problem due to offset needed to sign correctly the Cardano stake distribution. It will be merged shortly
- Refactor SlotNumber type #1851: the PR Replace SlotNumber alias with dedicated structure #1854 has been reviewed and merged. The issue is closed
- Slow import of Cardano transactions in testing-mainnet #1849: The issue is closed
- Benchmark signer footprint on SPO infrastructure #1826: we have started working on the issue, and the first developments have been pushed to the jpraynaud/1826-mithril-nodes-footprint branch. A PR will be created shortly
- Test Cardano transaction chain rollbacks #184: we are still monitoring the dev-mainnet network and we will complete the test shortly
- Upgrade Cardano node 9.1.0 #1855: the deployment of Cardano node 9.1.0 on thetesting-preview and testing-sanchonet has worked successfully. The issue is closed
- CIP for Mithril signature diffusion through Cardano network #1775: we will a draft branch before we create d draft PR on the CIP repository
- Finally, we have checked that none of our test SPOs needed KES key rotation

2024-07-25

Today, the team has worked on these topics:
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: the PR Implement signable and artifact builders for Cardano Stake Distribution #1847 is being updated before final review. It will be merged shortly
- Refactor SlotNumber type #1851: the PR Replace SlotNumber alias with dedicated structure #1854 has been created and is pending review. It will be merged shortly
- Slow import of Cardano transactions in testing-mainnet #1849: We have made another test of disk upgrade on the testing-mainnet network and have prepared the upgrade process
- Test Cardano transaction chain rollbacks #184: we have decided to run the test on the dev-mainnet network where a rollback is more likely to occur. We are monitoring and will complete the test shortly
- Upgrade Cardano node 9.1.0 #1855: the issue has been created and groomed. We have paired on the issue and we have created the PR Support Cardano node 9.1.0 #1856 has been created, reviewed and merged
- fix: clippy warnings with Rust 1.80.0 the PR has been created, reviewed and merged
- CIP for Mithril signature diffusion through Cardano network #1775: we will a draft branch before we create d draft PR on the CIP repository
- Finally, we have prepared the sprint planning for the next iteration

2024-07-24

Today, we have worked on these issues:
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: the PR Implement signable and artifact builders for Cardano Stake Distribution #1847 is being updated before we can merge it. We have investigated the source of some flakiness in the CI and we have worked in a fix that should be merged shortly
- Refactor SlotNumber type #1851: the issue has been created and we have started pairing on it. A PR will be created shortly
- ChainReader is stuck in Agency is theirs error #1836: the PR fix: ChainReader fails on find_intersect_point without agency #1852 has been reviewed and merged, which closes the issue
- Slow import of Cardano transactions in testing-mainnet #1849: the PR Upgrade infrastructure deployment with disk configurations #1850 as been updated as we have noticed some problem during a test upgrade on testing-mainnet. We will make another test shortly and complete the upgrade process
- Test Cardano transaction chain rollbacks #184: we are still waiting for a rollback to happen in the testing-preview. We will likely make the test on the dev-mainnet network where the probability of having a fork is higher
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP

2024-07-23

We have worked on these topics:
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: the PR Implement signable and artifact builders for Cardano Stake Distribution #1847 has been created and is pending review. We are investigating some flakiness in the CI and we will merge the PR as soon as it is fixed
- Refactor BlockNumber type #1755: the PR Replace BlockNumber alias with dedicated structure #1846 has been updated and merged. The issue is closed
- Refactor SlotNumber type #1851: the issue has been created and we have started pairing on it. A PR will be created shortly
- ChainReader is stuck in Agency is theirs error #1836: the PR fix: ChainReader fails on find_intersect_point without agency #1852 has been created and is pending review. It will be merged shortly
- Slow import of Cardano transactions in testing-mainnet #1849: we have created the PR Upgrade infrastructure deployment with disk configurations #1850 which extends the configuration of the disks in the terraform deployments. We have also bootstrapped a new dev-mainnet network to properly test the upgrade pbefore activating it in production
- fix: Resource not Reset when given back to Pool #1848: the PR has been reviewed and merged
- Test Cardano transaction chain rollbacks #184: we have started working on the issue and we should have results shortly
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Finally, we have held our weekly Office hours meeting in Discord

2024-07-22

We have worked on these topics:
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: the PR Implement signable and artifact builders for Cardano Stake Distribution #1847 has been created and is pending review. It will be merged shortly
- Refactor BlockNumber and SlotNumber types #1755: the PR Replace BlockNumber alias with dedicated structure #1846 has been created and is being reviewed. It will be merged shortly
- testing-sanchonet does not sign Cardano transactions #1845: we have investigated the problem and we have found that it is happening only on testing-sanchonet network. We have completed the issue with some explanation and we will optimize the chunk importer of the signer which is responsible for this problem in a separate issue. This issue is closed
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- ChainReader is stuck in Agency is theirs error #1836: we have started investigating the problem in the chain reader
- Slow import of Cardano transactions in testing-mainnet #1849: the issue has been created and investigated. We will keep working on it to upgrade the disks of the VMs
- fix: Resource not Reset when given back to Pool #1848: the PR has been created and is pending review. It will be merged shortly
- Finally, we have worked on the demo path for the monthly review meeting

2024-07-19

The team has worked on these issues:
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: we have kept pairing on the issue and developments have been pushed to the dlachaume/1832/implement-signable-artifact-builders-for-cardano-stake-distribution branch. A PR will be created shortly
- Refactor BlockNumber and SlotNumber types #1755: we have kept working on the issue. The developments ahve been published to the ensemble/1755/block_slot_number_type branch and a PR will be created shortly
- testing-mainnet and testing-sanchonet do not sign Cardano transactions #1845: the bug issue has been created and is being investigated
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- ChainReader is stuck in Agency is theirs error #1836: we are still investigating the bug
- We have also proceeded to the rotation of the KES keys of the test SPO running in the release-preprod network
- Finally, we have worked on the demo path for the monthly review meeting

2024-07-18

We have worked on these issues:
- Upgrade testing-sanchonet for respin with Cardano 9.0.0 #1822: the genesis certificate has been created and the network has resumed producing certificates. The issue is closed
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: we have kept working on the issue and developments have been pushed to the dlachaume/1832/implement-signable-artifact-builders-for-cardano-stake-distribution branch. A PR will be created shortly
- Refactor BlockNumber and SlotNumber types #1755: we have started working on the issue and a PR will be created shortly
- Make Certificate messages beacon field optional #1807: the PR Make deprecated beacon field optional in Certificate & CertificateList messages #1839 has been reviewed and merged. The issue is closed
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- ChainReader is stuck in Agency is theirs error #1836: we are still investigating the bug
- Test Cardano transaction chain rollbacks #1840: the issue has been created and groomed
- Cardano Stake Distribution Certification #955: the epic has been sliced and groomed
  - Implement routes for Cardano Stake Distribution in aggregator REST API #1841: the issue has been created and groomed
  - Implement sub-commands for Cardano Stake Distribution in client #1842: the issue has been created and groomed
  - Update explorer for Cardano Stake Distribution #1843: the issue has been created and groomed
  - Document Cardano Stake Distribution #1844: the issue has been created and groomed

2024-07-17

Today, the team worked on these topics:
- Cleanup Immutable File in Cardano transaction #1825: the PR cleanup immutable file in cardano transation #1838 has been created, reviewed and merged. The issue is closed
- Upgrade testing-sanchonet for respin with Cardano 9.0.0 #1822: the test SPO has successfully registered to the protocol and we should be able to create a genesis certificate tomorrow
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: we have kept working on the issue and developments have been pushed to the dlachaume/1832/implement-signable-artifact-builders-for-cardano-stake-distribution branch. A PR will be created shortly
- A recorded but non certified Cardano transaction creates an error in prover #1819: the issue has been fixed and closed
- Make Certificate messages beacon field optional #1807: we have paired on the issue and the PR Make deprecated beacon field optional in Certificate & CertificateList messages #1839 has been created and is pending review
- ChainReader is stuck in Agency is theirs error #1836: we are still investigating the bug
- Mithril signature diffusion with Cardano network layer PoC #1837: the epic has been created and groomed

2024-07-16

Today, the team has worked on these issues:
- Cardano transaction importer does not import the last block advertised as certified #1785: the PR Bugfix: Block range computation ignored last block range & last block range if it finished with empty blocks #1820 has been reviewed and merged, and the PR Make upper bound of ChainReaderBlockStreamer inclusive #1835 has been created, reviewed and merged. The issue is closed
- Non blocking artifact production in aggregator #1792: the PR non blocking artifact production in aggregator #1828 has been reviewed and merged. This closes the issue
- Upgrade testing-sanchonet for respin with Cardano 9.0.0 #1822: the testing-sanchonet has been respun, era markers have been set in the chain and the test SPO has been created. We need to wait for few epochs before the network resumes signing
- Document Prometheus metrics and Grafana Dahsboard for signer #1834: the issue has been created and groomed. The PR Add metrics table in the "Activate Prometheus endpoint" section #1833 has been created, reviewed and merged, and it closes the issue
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Implement signable and artifacts builders for Cardano Stake Distribution #1832: the issue has been created and groomed. We have started working on it and a draft PR will be created shortly
- ChainReader is stuck in Agency is theirs error #1836: the bug issue has been created. A fix PR will created shortly
- Release 2430 distribution #1830: the issue has been created and groomed
- We have also held our weekly office hours meeting on discord

2024-07-15

We have worked on these issues:
- Release 2428 distribution #1810: the distribution 2428.0 has been released and deployed on the release-mainnet and release-preprod networks
- Cardano transaction importer does not import the last block advertised as certified #1785: the PR Bugfix: Block range computation ignored last block range & last block range if it finished with empty blocks #1820 is being reviewed. It will be merged shortly
- Non blocking artifact production in aggregator #1792: we have kept pairing on the issue and the PR non blocking artifact production in aggregator #1828 has been created and is pending review. It will be merged shortly
- Upgrade testing-sanchonet for respin with Cardano 9.0.0 #1822: the PR Respin testing-sanchonet with Cardano 9.0.0 #1829 has been created, reviewed and merged. The testing-sanchonet network is being respun
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Benchmark signer footprint on SPO infrastructure #1826: the issue has been created and groomed

2024-07-12

Today the team worked on the following topics:
- Release 2428 distribution #1810: we have created the PR Rotate documentation for 2423 distribution #1732. It will be merged with the expected release of the new distribution next week
- Cardano transaction importer does not import the last block advertised as certified #1785: the PR Bugfix: Block range computation ignored last block range & last block range if it finished with empty blocks #1820 has been created and is pending review. It will be merged shortly
- Non blocking artifact production in aggregator #1792: we have paired on the issue and the developments have been pushed to the ensemble/1792/non-blocking-artifact-production-in-aggregator branch. A PR will be created shortly
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Calibrate Cardano transaction signature parameters #1814: the PR Configure Cardano transaction signing parameters with CI/CD #1815 has been reviewed and merged. The issue is closed
- Signer make test command fails #1816: the bug issue has been created. The PR Fix Signer make test command errors #1817 has been created, reviewed and merged, which closes the issue
- Finally, we have checked if some test SPOs needed to have their KES keys rotated, which was not the case

2024-07-11

We have worked on the following issues:
- Release 2428 distribution #1810: we have released the pre-release distribution 2428.0-pre which has been deployed to the pre-release-preview network
- Cardano transaction importer does not import the last block advertised as certified #1785: we are still investigating the problem. A new PR will be created shortly
- Non blocking artifact production in aggregator #1792: we have paired on the issue and the developments have been pushed to the ensemble/1792/non-blocking-artifact-production-in-aggregator branch. A PR will be created shortly
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Calibrate Cardano transaction signature parameters #1814: the issue has been created and groomed. The PR Configure Cardano transaction signing parameters with CI/CD #1815 has been created and is pending review

2024-07-10

Today, we have worked on these issues:
- Release 2428 distribution #1810: we have prepared the new distrobution pre-release with the PRs Upgrade dependencies #1811 and Update CHANGELOG with 2428 distribution release #1812 which have been created, reviewed and merged. We have also prepared some warmup node for fast switch to Cardano 9.0.0 on the pre-release-preview, release-preprod and release-mainnet networks
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Cardano transaction importer does not import the last block advertised as certified #1785: we are still investigating the problem. A new PR will be created shortly
- Non blocking artifact production in aggregator #1792: we have paired on the issue and the developments have been pushed to the ensemble/1792/non-blocking-artifact-production-in-aggregator branch. A PR will be created shortly
- Optimize Cardano transaction prover performances with parallelization #1756: the PR Parallelize Cardano Transaction prover #1781 has been closed as the performance were not improved by the parallelization. The issue is closed

2024-07-09

Today, we worked on these issues:
- Resource exhausted on Cardano node socket #1803: the bug is fixed and the issue is closed
- Certificate pending route overwhelms the Cardano node in aggregator #1804: the PR Remove call to chain observer on pending certificate route #1808 has been created, reviewed and merged. The issue is closed
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Cardano transaction importer does not import the last block advertised as certified #1785: we are still investigating the problem. A new PR will be created shortly
- Upgrade Cardano node 9.0.0 #1787: the PR Refactor: Cardano node configuration in infra #1806 has been reviewed and merged. The PR Support Cardano node 9.0.0 #1809 has been created, reviewed and merged. The issue is closed
- Non blocking artifact production in aggregator #1792: we have paired on the issue and the developments have been pushed to the ensemble/1792/non-blocking-artifact-production-in-aggregator branch. A PR will be created shortly
- Release 2428 distribution #1810: the issue has been created and groomed

2024-07-08

Today, the team has worked on the following topics:
- Resource exhausted on Cardano node socket #1803: the bug is being fixed
- Certificate pending route overwhelms the Cardano node in aggregator #1804: we have paired on the issue and the developments have been pushed to the ensemble/1804/remove_call_to_chain_observer_on_pending_certificate_route branch. A PR will be created shortly
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the redaction of the draft CIP
- Cardano transaction importer does not import the last block advertised as certified #1785: we have kept working on the issue has the problem is not completely fixed
- Upgrade Cardano node 8.12.2 #1787: the PR Refactor: Cardano node configuration in infra #1806 has been created and is pending review. It will be merged shortly
- Make Certificate messages beacon field optional #1807: the issue has been created and groomed
- UpkeepService tests flakiness #1802: the PR has been created, reviewed and merged

2024-07-05

Today, the team has worked on these issues:
- Resource exhausted on Cardano node socket #1803: the bug issue has been created and investigated
- Certificate pending route overwhelms the Cardano node in aggregator #1804: the issue has been created
- CIP for Mithril signature diffusion through Cardano network #1775: we have kept working on the modeling of the network cost and the redaction of the n2n mini protocol in the draft CIP
- Finally, we have checked if some test SPOs needed to have their KES keys rotated, which was not the case

2024-07-04

Today, the team has worked on these issues:
- Upgrade Cardano node 8.12.2 #1787: the PR Fix: Support Cardano 8.12 #1799 has been created, reviewed and merged. We now need to find a way to handle properly the mainnet configuration for 8.12 which is still labeled as pre-release in the Cardano books. We will create a PR shortly
- Cardano transaction importer does not import the last block advertised as certified #1785: the PR Bugfix: Can't generate proof for transactions on the last signed block number #1795 has been reviewed and merged. We are monitoring the results on the test networks to make sure the problem is fixed
- Aggregator/Signer preload transactions when Cardano transactions certification is not activated #1782: the PR Preload Cardano transactions based on Aggregator's Signing Capability #1796 has been reviewed and merged. The issue is closed
- Lint Markdown/JavaScript files in repository #1754:the PR Standardize Javascript and Markdown code formatting #1783 has been reviwed and merged. The issue is closed
- Optimize Cardano transaction prover performances with parallelization #1756: the draft PR has been updated and will be ready for review shortly
- fix: macOS e2e local tests fails on M3 silicon processors #1767: the PR has been reviewed and needs to be updated to address the review comments

2024-07-03

Today, we have worked on the following topics:
- Upgrade Cardano node 8.12.2 #1787: we have discovered many problems with the activation of 8.12 and we have worked on it today. The testing-preview network is working but needs another fix on the used configurations. The testing-sanchonet network has been rolled back to 8.11.0-sancho
- Cardano transactions import blocks aggregator and signer #1797: the bug issue has been created and the PR Fix: aggregator and signer blocked during Cardano transactions import #179 has been created, reviewed and merged. The issue is closed
- Cardano transaction importer does not import the last block advertised as certified #1785: the PR Bugfix: Can't generate proof for transactions on the last signed block number #1795 has been created and is pending review. It will be merged shortly
- Aggregator/Signer preload transactions when Cardano transactions certification is not activated #1782: the PR Preload Cardano transactions based on Aggregator's Signing Capability #1796 has been created and is pending review. It will be merged shortly
- SQLite WAL files are not truncated in signer and aggregator #1707: the issue is closed
- fix: macOS e2e local tests fails on M3 silicon processors #1767: the PR has been reviewed and needs to be updated to address the review comments
- Lint Markdown/JavaScript files in repository #1754: we have created the draft PR Standardize Javascript and Markdown code formatting #1783. It will be merged shortly
- Optimize Cardano transaction prover performances with parallelization #1756: the draft PR has been updated and will be ready for review shortly

2024-07-02

Today, we have worked on the following issues:
- Upgrade Cardano node 8.12.2 #1787: the PR Support Cardano node 8.12.2 #1788 has been reviewed and merged
- SQLite WAL files are not truncated in signer and aggregator #1707: the PR Upkeep service for aggregator & signer #1791 has been reviewed and merged
- Cardano signatures are not produced on testing-sanchonet #1750: the PR Fix sanchonet blocks parsing #1794 has been created, reviewed and merged. The testing-sanchonet has resumed certification of the Cardano transactions. The issue is closed
- fix: macOS e2e local tests fails on M3 silicon processors #1767: the PR has been reviewed and needs to be updated to address the review comments
- Lint Markdown/JavaScript files in repository #1754: we have created the draft PR Standardize Javascript and Markdown code formatting #1783. It will be merged shortly
- Optimize Cardano transaction prover performances with parallelization #1756: the draft PR has been updated and will be ready for review shortly
- Cardano transaction importer does not import the last block advertised as certified #1785: we have kept working on the issue and the developments have been pushed to the djo/1785/last_tx_not_proved branch
- Aggregator/Signer preload transactions when Cardano transactions certification is not activated #1782: the issue has been groomed. We have started pairing on it. The developments have been pushed to the ensemble/1762/preload-cardano-ctx-when-ctx-not-activated branch. A PR will be created shortly
- Additionally, we have held our weekly Office Hours meeting on Discord

2024-07-01

Today, the team has worked on the following issues:
- Upgrade Cardano node 8.12.1 #1787: the PR Support Cardano node 8.12.1 #1788 has been updated. It will be ready for review shortly
- Conditional embedding of Cardano CLI in Docker images #1725: the PR Conditional embedding cardano cli in docker image #1789 has been reviewed and merged. The issue is closed
- SQLite WAL files are not truncated in signer and aggregator #1707: we have created the PR Upkeep service for aggregator & signer #1791 which is pending review
- Cardano signatures are not produced on testing-sanchonet #1750: the PR fix: fix CBOR decoding of Conway params update #473 has been created on pallas repository and will be merged shortly. When the 0.27.1 patched version is released, we will create a PR with the fix
- fix: macOS e2e local tests fails on M3 silicon processors #1767: the PR has been reviewed and needs to be updated to address the review comments
- Lint Markdown/JavaScript files in repository #1754: we have created the draft PR Standardize Javascript and Markdown code formatting #1783. It will be merged shortly
- Fix aggregator Cardano transaction prover benchmark tool #1790: the PR has been merged
- Optimize Cardano transaction prover performances with parallelization #1756: the draft PR has been updated and will be ready for review shortly
- Non blocking artifact production in aggregator #1792: the issue has been created and groomed
- Explorer does not handle invalid transaction hashes #1784: the issue has been groomed
- Resource Pool acquire timeout when being updated #1793: the issue has been created and groomed