ic-stable-memory Library Introduction

Library's repository: github.com/seniorjoinu/ic-stable-memory

What's wrong with stable memory

If you're an IC canister developer and you use Rust, then I'm sure you've asked yourself the following question:

"Ok, my canister has 4GB of heap memory and 8GB of stable memory, but how do I actually use these 8GB? The only thing I can do is to serialize my heap (and, in practice, you are only able to serialize 2GB max because of gas limits) in pre_upgrade() and then deserialize it back in post_upgrade(). So, even if I could somehow save more that 4GB of my heap to stable memory, there is no way I can get it back - it just won't fit."

And this problem exists because of the way we serialize and store our data to stable memory. Candid serialization is not a good way for storing something in stable memory, since it is intended for message encoding - for collecting different pieces of data into a single message, which will be sent over the network. And, at the moment, there is no way of getting an individual piece of data from a serialized message without deserializing the whole message first.

With stable memory, we want something different. We want these different pieces of data to be accessible independently from each other, so we're not forced to spend a lot of cycles deserializing gigabytes of data each time we want to read only a couple of kilobytes from it.

Okay, let's imagine that now we're storing not a single huge blob of bytes with all the data serialized in it, but many small blobs each of which we can access individually. Now we have a whole bunch of new problems:

How do we address these blobs?
How do we allocate new stable memory for them?
How do we de-allocate stable memory, when we no longer need this data?
How do we re-allocate the memory, when the data that is stored in it changed and became bigger?
How do we make collections of such data (a simple vector, for example)?
How do we make all of this work, but keep the developer experience safe and enjoyable?

So, ic-stable-memory is a library, that handles all of that, so you could finally use all of your canister's memory.

How does it look

Okay, we want to save some data to stable memory, how do we do that? Instead of saving it to some heap-living data structure (e.g. a static variable) and then serializing/deserializing it during pre_upgrade/post_upgrade hooks, we'll store it directly to stable memory and we'll only handle the pointer to it during pre_upgrade/post_upgrade.

In ic-stable-memory, when we want to store something, we define a stable variable for this data:

type MyVariable = i32;

s! { MyVaraible = 10 };

Note that for any data we want to store in stable memory, we have to define a type alias. This type alias then will be used as a unique identifier for our stable variable that helps the s! macro to typecheck the data and prevents us from making typos. Look at this as on a new syntax for defining variables: first we define a variable name and the type with typealias and then we use this variable name to write data to it or read data from it.

And this is it. We just created a stable variable named MyVariable
that will hold an i32 value of 10.

In order to update this stable variable we just need to call s! macro once more:

s! { MyVariable = 999 };

In order to read a stable variable, we use a simplified version of s! macro:

let my_variable = s!(MyVariable);

assert_eq!(my_variable, 999); // true

And... this is it. You can create a bunch of such stable variables and store any data inside, not just numbers. ic-stable-memory uses Speedy serialization library. So if you want to save some complex data to a stable variable, you have to make it speedy-serializable by implementing speedy::Readable and speedy::Writable.

This data already lives in stable memory and you don't have to serialize and deserialize it before and after canister upgrades. The only thing you have to do is to call a couple of library's functions so it could handle its own internal metadata between upgrades:

#[init]
fn init() {
    stable_memory_init(true, 0);
}

#[pre_upgrade]
fn pre_upgrade() {
    stable_memory_pre_upgrade();
}

#[post_upgrade]
fn post_upgrade() {
    stable_memory_post_upgrade(0);
}

This routine is very cheap - it only stores some internal pointers, nothing more.

Collections

Okay, we can store relatively small data structures in such variables, but what if we have like a huge (gigabytes in size) data collection? Maybe some huge map of accounts or a log of historical events. In this case there are stable collections available. For now there are only five of them:

a stable vector SVec which is indexed with u64;
a stable hash map SHashMap which is indexed with any data type that implements Hash and Eq;
a stable btree map SBTreeMap, which is indexed with any data type that implements Ord;
a stable hash set SHashSet, which stores any data type that implements Hash and Eq;
a stable btree set SBTreeSet, which stores any data type that implements Ord;
a stable binary heap SBinaryHeap, which stores any data type that implements Ord.

All these collections are able to hold and efficiently provide an access to as many entries as much stable memory the subnet can allocate to your canister. APIs of these collections are very basic at the moment, but they get their jobs done. For example, this is how we can use SVec in order to store some historical events:

type MyHistory = SVec<HistoryEntry>;

#[derive(Readable, Writable)]
struct HistoryEntry {
    pub data: String,
    pub timestamp: u64,
}

#[init]
fn init() {
    ...
    
    // initialize the stable collection
    s! { MyHistory = SVec::<HistoryEntry>::new() };

    ...
}

#[update]
fn add_history_entry(data: String) {
    let entry = HistoryEntry {
        data,
        timestamp: time()
    };

    // retrieve the pointer to SVec
    let mut my_history = s!(MyHistory);
    
    // push a new entry
    my_history.push(&entry);

    // update the stable variable
    s! { MyHistory = my_history };
}

#[query]
fn get_history_entry(id: u64) -> HistoryEntry {
    let my_history = s!(MyHistory);

    // unwrapping, since it returns an Option
    my_history.get_cloned(&id).unwrap() 
}

What's different with these collections is that they store their data directly in stable memory. When we're retrieving such a collection we're only retrieving it's metadata, but not the actual content that is stored inside. The content is retrieved only when we explicitly ask for it with get_cloned() function.

Out of memory

There is one more thing. With ic-stable-memory you're not only able to store data directly in stable memory, but to react when there is no more free memory in your canister.

In order to do that, you have to specify a special update function in your canister:

#[update]
async fn on_low_stable_memory() {
    // example
    scale_horizontally().await;
}

This function should be named on_low_stable_memory(), should have no arguments and no return values. You can do whatever you want inside it: from e-mailing the developer to spawning a new canister to scale horizontally and be able to hold more data.

This function will be called when your canister (read, the subnet your canister in) is almost out of memory. Your canister will continue to work as expected after the invocation, but after some time it will no longer be able to hold more data (unless you remove some of it).

Conclusion

This tutorial is pretty short and straightforward and this is actually great for the introduction - it means that ic-stable-memory is easy to use. But the insides of this library are quite complex. There is another article for those readers who want to peek on what is under the hood and how it works.

Anyway, I hope you found this article interesting. I would love to see your feedback on this library, so please, try it out.

Take care.

​ic-stable-memory Library Introduction

​What's wrong with stable memory

​How does it look