mapage

  • 2nd Aug 2023
  • 9 min read
  • Last updated on 25th Jan 2024

Mapage

Mapage is a type-namespaced key-value storage system which uses async-graphql for client interaction.

Basically what is meant by "type-namespaced" is each storable type gets its own hashmap and its own part of the API.


Design Goals

  • Customisable: Choose which types you want to store and which map implementations you want store them in.
  • Easily-Optimisable: Easily find the best features for the fastest results.
  • Type-Namespaced: Incorrect type errors should be few and far between, or non-esitant when you specify exacty which types you want stored.
  • file storage: Chache files with ease.
  • Persistable: Store any changes to your chached data in a database system of your choice or on the file .
  • Scriptable: Implement the funtionality of your choice.
  • Monitorable: Easily find out things like CPU and RAM usage.
  • Searchable: From rudimentary to regex to vector based searching functionality.

Atomicity Levels

The namespaced-type design makes it reasonably straightforward to support different atomicity levels:

LevelDescription
StoreOnly one action (read or write) can be performed on the store per interaction.
NamespaceOnly one action can be performed on each individual namespace per interaction.
Sub-NamespaceA section of the namespace is locked per interaction (buckets)
ValueEach value is interacted with by each thread on an individual level and the namespace is seldom if ever locked

In terms of features atomicity levels are divided into two broad categories:

Store level Feature
store_aml
sub_store_aml

With store_aml (store atomicity level) all the data is contained in a single synchronisation object e.g a Mutex or an Actor, but with the sub_store_aml (sub-store atomicity level) feature you have more discretion about how each namespace goes about handling user interaction.

store_aml is not implemented.

Once you've selected sub_store_aml you must select a default sore implementation feature:

Default Storage Implemtation Feature
scc_hashmap_namespaces
dashmap_namespaces

As you can probably sermise; when you select "scc_hashmap_namespaces" you get scc::Hashmap namespace implementations by default and when you select "dashmap_namespaces" you get Dashmap namespace implementations by default.

Select only one of these.


All Features

FeatureDescription
store_amlStore atomicity level
sub_store_amlSub-store atomicity level
scc_hashmap_namespacesUse scc::Hashmap namespace implementations by default
dashmap_namespacesUse Dashmap namespace implementations by default
all_typesAll types
charchar type
f32f32 type
f64f64 type
i8i8 type
i16i16 type
i32i32 type
i64Broken - incompatible with GraphQL/async-graphql
i128Broken - incompatible with GraphQL/async-graphql
isizeTo be removed
StringString type
u8u8 type
u16u16 type
u32u32 type
u64Broken - incompatible with GraphQL/async-graphql
u128Broken - incompatible with GraphQL/async-graphql
usizeTo be removed
WhateverA GraphQL union of everything except SelectedType
SelectedTypeA GraphQL union of whatever else was selected except Whatever - To be removed
Vec_boolA std::vec::Vec of bools
Vec_char     "          chars
Vec_f32     "          f32s
Vec_f64     "          f64s
Vec_i8     "          i8s
Vec_i16     "          i16s
Vec_i32     "          i32s
Vec_i64Broken - incompatible with GraphQL/async-graphql
Vec_i128Broken - incompatible with GraphQL/async-graphql
Vec_isizeTo be removed
VecSelectedTypeTo be removed
VecStringA std::vec::Vec of std::string::Strings
Vec_u8     "          u8s
Vec_u16     "          u16s
Vec_u32     "          u32s
Vec_u64Broken - incompatible with GraphQL/async-graphql
Vec_u128Broken - incompatible with GraphQL/async-graphql
Vec_usizeTo be removed
VecWhateverA std::vec::Vec of Whatevers
all_key_types_StringUse std::string::String keys by default
all_key_types_Arc_StringUse std::sync::Arc containing std::string::String keys by default

Settings

To be implemented


Build Examples

One with everything:

cargo build --features=sub_store_aml,scc_hashmap_namespaces,all_types,all_key_types_String

In the above build you have the "sub_store_aml" feature to indicate that each namespace handles synchronisation independently.

The "scc_hashmap_namespaces" feature indicates that you want the namespace implemtations to be scc::Hashmaps by default.

The "all_types" feature indicates that you want all the types that mappage supports to be inculded in the build.

Finally the "all_key_types_String" feature indicates that you want the keys for all the included types to be std::string::Strings by default.


Using the all_types feature, while convenient, will produce a mapage application with gigantic GraphQL API.

Why don't we try and be more a bit more selective:

cargo build --features=sub_store_aml,dashmap_namespaces,bool,char,i32,Whatever,all_key_types_String

Here instead of sub_store_aml, we have dashmap_namespaces for the namespace implementations.

And instead of all_types there are four individual types specified: bool, char, i32 and Whatever.


When you know what you want to store you'll avoid using space unnecessary and have a more consise API than you would've had selecting everything available.


Build Must-Haves

To be explicit, in every build you need or should have specified:

  1. The store atomicity level (only sub_store_aml is valid right now)
  2. The namespace implementations for each selected type to use by default.
  3. The types you want (char, f32, i8 etc) (this is optional but you wont have much of a cache if you skip this step)
  4. The key type to use for each namespace by default

Feature incompatabilty error conditions have not be implemented yet so you may have to do some reading if you get something wrong.

Optimal Builds

To be implemented


A note on experimentation

While you would implement the most optimal features for practical usage scenarios, sub-optimal features will also be added for curiositys sake.


Todo:

  • Add store/namespace management features which would handle:
    • Object evictions.
    • Authentication and authorisation
    • Persistence
    • Replication (partial and whole)
    • Events/Streaming
  • Add sub-namespaces
  • Add immutable namespaces, where the values can be added, retrieved, removed, replaced, have only immutable methods called on them. Also this is where collections are put in Arcs.
  • Async-graphql response caching (possibly in 5 second increments)
  • Scripting with Rhai and Lua (others?)
  • WGPU integration - to be used in conjunction with scripting.
  • Arc'd String Keys and other key types
  • Add error conditions and messages for conflicting feature specifications.
  • Vector comparison functionality
  • Vector search functionality
  • Add more namespace implementations using collection types like Moka and std::Hashmap etc.
  • Complete namespace implementations for the store and atomicity levels.
  • Add regex storage
  • Add the ability to search for namespace items using a regex that is either stored or part of the request i.e. Mass deletion based on whether on not a given regex matches etc.
  • Add the ability to remove namespace items based on weather or not it starts with or ends with a certain sequence of values or otherwise contains this sequence of values.
  • Clean up the code.
  • Document the code.

Possible:

  • Option Types (Unlikely, unless it's Vec<Option<T>>)

Known Issues And Types To Be Removed Or Disabled

FeatureIssue
i64Broken - incompatible with GraphQL/async-graphql
i128Broken - incompatible with GraphQL/async-graphql
isizeTo be removed
u64Broken - incompatible with GraphQL/async-graphql
u128Broken - incompatible with GraphQL/async-graphql
usizeTo be removed
SelectedTypeA GraphQL union of whatever else was selected except Whatever - To be removed/disabled
Vec_i64Broken - incompatible with GraphQL/async-graphql
Vec_i128Broken - incompatible with GraphQL/async-graphql
Vec_isizeTo be removed
VecSelectedTypeTo be removed/disabled
Vec_u64Broken - incompatible with GraphQL/async-graphql
Vec_u128Broken - incompatible with GraphQL/async-graphql
Vec_usizeTo be removed

So GraphQL doesn't support intergers with sizes above 32 bits.

I'll implement a work around for this issue.

The isize and usize types are redundant.

SelectedType was implemented for an experiment in asynchronous queries.

I think I got to the trial implementation stage with this feature and I may revisit it so I'll remove SelectedType and VecSelectedType in the next version but keep the code in the repository.


About Generic Permutations

In order to control the amount of work requred certain classes of type will and won't be implemented:

  • No option unless in other gernerics e.g. Vec<Option<T>>. While having a base level Option could be a convenient way to reserve a key in a namespace, for the time being at least, base level Option namespaces wont be added.
  • With hashmaps and other generic types that take multiple augments you can have all manner of different combinations. So the keys of namespace implementations will probably be constrained to what you can select with the available features and not arbitrary values (sub-namespaces).

This project has come along way since its inception in mid-2022 as a Redis-like cache and it has a long way to go still. Please use with the understanding that this project is a work in progress.


Coding Style

The coding style of this project emphasises the use of white space over keeping the line and column counts low.

So this:

fn foo()
{

    bar();

}

Not this:

fn foo()
{
    bar();
}


License

Licensed under either of:

  • Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0 (see also: https://www.tldrlegal.com/license/apache-license-2-0-apache-2-0))
  • MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT (see also: https://www.tldrlegal.com/license/mit-license))

at your discretion


Contributing

Please clone the repository and create an issue explaining what feature or features you'd like to add or bug or bugs you'd like to fix and perhaps how you intend to implement these additions or fixes. Try to include details though it doesn't need to be exhaustive and we'll take it from there (dependant on availability).


Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.