Hello,Please find the cip-0049 open for discussion at the PR https://github.com/global-synchronizer-foundation/cips/pull/31In order to move to a vote we are requesting a Super Validator to sponsor the CIP and a second Super Validator to endorse the CIP.Any SV sponsors/endorses or changes needed?Dr. Amanda L. Martin
814-359-6544Director of Program Management
Could we also define what we mean by cold? Are we talking about fully air gapped cold storage? Same question as Eric – how often?
Is there a forum to discuss this CIP?
Kinga Z. Bosse
COO|MPCH|www.mpch.com
|
|
Hello,
Please find the cip-0049 open for discussion at the PR https://github.com/global-synchronizer-foundation/cips/pull/31
In order to move to a vote we are requesting a Super Validator to sponsor the CIP and a second Super Validator to endorse the CIP.
Any SV sponsors/endorses or changes needed?
Dr. Amanda L. Martin
814-359-6544Director of Program Management
--
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to
important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html.
If you are not the intended recipient, please delete this message.
A regular process for DR testing like how the Fia does it might be useful. https://www.fia.org/fia/fia-disaster-recovery-exercise.
Hello,
Please find the cip-0049 open for discussion at the PR https://github.com/global-synchronizer-foundation/cips/pull/31
In order to move to a vote we are requesting a Super Validator to sponsor the CIP and a second Super Validator to endorse the CIP.
Any SV sponsors/endorses or changes needed?
Dr. Amanda L. Martin
814-359-6544Director of Program Management
--
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to
important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html.
If you are not the intended recipient, please delete this message.
While some of the technical details are discussed I wanted to also raise a tokenomics questions. This line from the CIP could use some additional detail.
>> Super Validators that maintain and verifiably prove the integrity of their backups will receive an enhanced reward, equivalent to an approximate bonus of 20–30% (in Canton terms, a +1 weight boost on current rewards).
Specifically, is it a bonus of 20-30% or a bonus of +1 weight boost? If it is in absolute terms, the award for a T3 SV would be 100% and for a T1 SV would be 10%. Are we
looking to make the reward consistent in absolute terms or in relative terms? If we were to do something here, it needs to be configured thoughtfully. For instance, could a T4 0.5 weight SV add 200% to its weight by just running a backup?
Regardless of what is being proposed here, as with any CIP that is tokenomics related, the structure needs to be precise and leave no room for interpretation.
A regular process for DR testing like how the Fia does it might be useful. https: //www. fia. org/fia/fia-disaster-recovery-exercise. From: cip-discuss@ lists. sync. global <cip-discuss@ lists. sync. global> on behalf of Eric Saraniecki via lists. sync. global
A regular process for DR testing like how the Fia does it might be useful. https://www.fia.org/fia/fia-disaster-recovery-exercise.
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
Yiannis,This could've been an option however there are some challenges:1) there should be a way to restore data from the network inception till one month before current time without gaps2) catching up could be pretty slow (something like 1/10 of real time)Another option could be picking a few arbitrary (somewhat random) points in time, fetching states at those points from the archive and comparing them to some references. This is also not without challenges however this is probably somewhat close to how the archive would be used when it's needed.Stas
Chris, you are right - this is very ambiguous and I realised that It's not complete.The idea was to have a relative bonus model of 20% with a cap of +1 in the weight, whatever is smaller.No validator should exceed a +1 weight increase by gaining 20% more rewards.
To break it down with an example, with relative bonus of 20%, a SV with a weight of 10 would
be getting the equivalent rewards of a having a weight of 12 which is way to much
for maintaining a backup. So I propose to have a cap of +1 in weight equivalent in termsof the rewards. So they will be getting a 10% bonus instead.
Another solution - that now that I think about it I like more - is to have a simplereward pool of X canton coins, that is split equally between the SV thathold and verify the backups. This simplifies the logistics a lot since thereis no linear increase in rewards and it's irrelevant from the weights, so more fair.
The parameters I propose here are not set. Everyone is runninga different setup in terms of costs. We need to incentivise enough in order to have ALL SV keepbackups but not to the point that the network token economy is affected.
I strongly suggest that we focus on formalising the backup and verification process first.Then, we will have a good idea of what the costs for running this network-wide is.Last step is to formalise the economic incentives.
Y.
Can you please elaborate on this? My understanding is that we can query specific data timestamps.
thanks.
Not to distract the conversation, but wanted to close the loop re: being more precise on the weight changes associated with this.
In short, I’m ok with focusing on the technical implementation details first and re-visiting the incentive construct later. I like the idea of uses caps to address corner cases. I’ll give some thought to the question of weight increase vs. token award pool but my lean is to weight increases for now.
Chris
> Can you please elaborate on this? My understanding is that we can query specific data timestamps. From a live system we can definitely query specific data timestamps. But here we're talking about database snapshots kept in cold storage.
> Can you please elaborate on this? My understanding is that we can query specific data timestamps.
From a live system we can definitely query specific data timestamps. But here we're talking about database snapshots kept in cold storage. These are static blobs of data, often in binary format, with no supporting live system to query from.
At least my understanding of the proposed CIP is not to keep this data live in the system. We will still prune it from the live system, so it cannot easily be queried by hitting some endpoint. We will only keep historical snapshots of the databases in cold storage.
Itai
On Wed, Mar 5, 2025 at 12:31 AM Yiannis Varelas via lists.sync.global <y=fivenorth.io@...> wrote:
Can you please elaborate on this? My understanding is that we can query specific data timestamps.
thanks.
--
Itai Segall
Director of Engineering, Canton Network / +1 551 208 8835
Digital Asset, creators of Daml
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to
important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html.
If you are not the intended recipient, please delete this message.
> Can you please elaborate on this? My understanding is that we can query specific data timestamps.From a live system we can definitely query specific data timestamps. But here we're talking about database snapshots kept in cold storage. These are static blobs of data, often in binary format, with no supporting live system to query from.At least my understanding of the proposed CIP is not to keep this data live in the system. We will still prune it from the live system, so it cannot easily be queried by hitting some endpoint. We will only keep historical snapshots of the databases in cold storage.ItaiOn Wed, Mar 5, 2025 at 12:31 AM Yiannis Varelas via lists.sync.global <y=fivenorth.io@...> wrote:Can you please elaborate on this? My understanding is that we can query specific data timestamps.
thanks.--
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
Hi Itail,I think we need to develop a backup prover. The prover can be run by any SV. Periodically the prover will raise a challenge to ask among its peers for some pieces of data. The prover will need to figure out how to get that data and respond to that challenge.To support different backup solutions, we can implement a plug-in architecture similar to terraform, where terraform provider is a separate program that runs and communicates with the main terraform through rpc call. we can provide a few provider providers such as restic backup, borg backup, ebs snapshot etc.There are 2 kinds of data in our specific setup1. Level DB dataThis is straightforward to easily mount a backup snapshot into disk, and have a simple Go program that reads the key and returns its value. Or restore an EBS snapshot.
CometBFT, currently in our config, uses this https://github.com/syndtr/goleveldb which is fairly simple to write a small program that opens the data (either from snapshot mount through FUSE or an EBS volume) and fetch the key.2. Postgres DataSimilarly as above, with Postgres data, the prover can restore an instance from their snapshot data. Then it can query for the relevant data, and then shutdown the postgres instance. For non-cloud backup solutions like pgbackrest or pgbarman, we can pick one and write a provider to handle restoration, read the data out.For picking a data point out to compare, the prover can try to pick the one that has not changed during the backup window. Example we pick a key or a row that has not changed in 24hours, then regardless which time the backup is taken during the day, it should have that value.What do you think ?On Wed, Mar 5, 2025 at 5:48 AM Itai Segall via lists.sync.global <itai.segall=digitalasset.com@...> wrote:> Can you please elaborate on this? My understanding is that we can query specific data timestamps.From a live system we can definitely query specific data timestamps. But here we're talking about database snapshots kept in cold storage. These are static blobs of data, often in binary format, with no supporting live system to query from.At least my understanding of the proposed CIP is not to keep this data live in the system. We will still prune it from the live system, so it cannot easily be queried by hitting some endpoint. We will only keep historical snapshots of the databases in cold storage.ItaiOn Wed, Mar 5, 2025 at 12:31 AM Yiannis Varelas via lists.sync.global <y=fivenorth.io@...> wrote:Can you please elaborate on this? My understanding is that we can query specific data timestamps.
thanks.--
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
I don't want to make it too complicated, but there are "proof of storage" systems for e.g. filecoin, which I believe solve this problem. I'd be happy to look into this further if there is interest.
Thanks Vinh,
At a high level, that makes perfect sense and I've been toying around with the same idea in my mind over the last few days.One thing I currently do not have an answer for, which might be a blocker, is how does the challenger know whether the answer is correct. In order for this to be something that minting depends on, you would need somehow to get 2/3 of SVs to attest that they accept the answer.
Then there's some interesting challenges around who operates the challengers itself, how do SVs submit their responses without revealing the answer to others, etc. but those can probably be resolved fairly easily.
Show quoted message
On Fri, Mar 7, 2025 at 8:51 AM Vinh Nguyen via lists.sync.global <v=fivenorth.io@...> wrote:
Hi Itail,
I think we need to develop a backup prover. The prover can be run by any SV. Periodically the prover will raise a challenge to ask among its peers for some pieces of data. The prover will need to figure out how to get that data and respond to that challenge.
To support different backup solutions, we can implement a plug-in architecture similar to terraform, where terraform provider is a separate program that runs and communicates with the main terraform through rpc call. we can provide a few provider providers such as restic backup, borg backup, ebs snapshot etc.
There are 2 kinds of data in our specific setup
1. Level DB data
This is straightforward to easily mount a backup snapshot into disk, and have a simple Go program that reads the key and returns its value. Or restore an EBS snapshot.
CometBFT, currently in our config, uses this https://github.com/syndtr/goleveldb which is fairly simple to write a small program that opens the data (either from snapshot mount through FUSE or an EBS volume) and fetch the key.
2. Postgres Data
Similarly as above, with Postgres data, the prover can restore an instance from their snapshot data. Then it can query for the relevant data, and then shutdown the postgres instance. For non-cloud backup solutions like pgbackrest or pgbarman, we can pick one and write a provider to handle restoration, read the data out.
For picking a data point out to compare, the prover can try to pick the one that has not changed during the backup window. Example we pick a key or a row that has not changed in 24hours, then regardless which time the backup is taken during the day, it should have that value.
What do you think ?
On Wed, Mar 5, 2025 at 5:48 AM Itai Segall via lists.sync.global <itai.segall=digitalasset.com@...> wrote:
> Can you please elaborate on this? My understanding is that we can query specific data timestamps.
From a live system we can definitely query specific data timestamps. But here we're talking about database snapshots kept in cold storage. These are static blobs of data, often in binary format, with no supporting live system to query from.At least my understanding of the proposed CIP is not to keep this data live in the system. We will still prune it from the live system, so it cannot easily be queried by hitting some endpoint. We will only keep historical snapshots of the databases in cold storage.
Itai
On Wed, Mar 5, 2025 at 12:31 AM Yiannis Varelas via lists.sync.global <y=fivenorth.io@...> wrote:
Can you please elaborate on this? My understanding is that we can query specific data timestamps.
thanks.
--
Itai Segall
Director of Engineering, Canton Network / +1 551 208 8835Digital Asset, creators of Daml
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
--
Itai Segall
Director of Engineering, Canton Network / +1 551 208 8835Digital Asset, creators of Daml
This message, and any attachments, is for the intended recipient(s) only, may contain information that is privileged, confidential and/or proprietary and subject to important terms and conditions available at http://www.digitalasset.com/emaildisclaimer.html. If you are not the intended recipient, please delete this message.
Hi all - just bringing this up again since we have now gone through the devnet reset.@Wayne I believe 2/3 is what we need yes.@Ryan this is an excellent idea. Had a look at filecoin and seems that they spin outa business that does exactly that.Might worth looking into this as a complete solution.
As discussed in the last weekly ops meeting, I would like to share a few approaches to bring this up for discussion again.In the new CometBFT we are going to use Postgres as storage instead of LevelDB so we can focus all the effort on proving Postgres data. If we do need to perform that, LevelDB is actually quite easy to prove because they are just key<-> values with limited functionality and can easily restore a snapshot and mount. Backup tooling like `restic snapshot` even allow to mount a snapshot and we can read the leveldb file with any client example https://github.com/liderman/leveldb-cli/blob/master/main.goFor Postgres, the primary challenge is access postgres backup data in a timely manner without restoring it into a real database.
There are however many modern Postgres tools that allow us to export Postgres data as CSV or Parquet file to object storage and allow us to query these directly using SQL interfaces without the need to restore the data to a Postgres instance.
The process is very similar to a full pg_dump, but instead of exporting to CSV/SQL or custom format, we export the data to Parquet files, and the file can be stored on object storage like S3/GCS.
Example tooling: https://github.com/CrunchyData/pg_parquet/ https://bnmoch3.org/notes/2025/postgres-iceberg-sync-duckdb/These parquet files can then be query directly using SQL with tool such as AWS Athena(https://docs.aws.amazon.com/athena/latest/ug/columnar-storage.html), Druio, DuckDB(https://duckdb.org/docs/stable/guides/network_cloud_storage/s3_import.html)We then can write a service that performs SQL queries to verify data consistency between the backup. If we take backup daily, we will need to craft a query that has not been modified in say the last 24 hours, then no matter when a backup was taken, as long as it's in the 24 hours window, the backup is correct and will be consistent across all SV.Example, today is 2025-04-15. We can ask "give me a list of transfers between 2025-04-13 -> 2025-04-14. Then no matter when the backup is taken on 2025-04-15, the data will be consistent.Let me know if this idea is sound and we can expand further into some real http service that can perform this kind of query.On Wed, Mar 19, 2025 at 9:31 AM Yiannis Varelas <y@...> wrote:Hi all - just bringing this up again since we have now gone through the devnet reset.@Wayne I believe 2/3 is what we need yes.@Ryan this is an excellent idea. Had a look at filecoin and seems that they spin outa business that does exactly that.Might worth looking into this as a complete solution.
