• DevelopersGuide
• DataStore

DataStore Server

The MINIX 3 DataStore (DS) server allows components to back up data and retrieve it after a restart and allows them to exchange data indirectly. For each publish request, an entry is stored in DS.

Currently DS supports 5 types of data: U32 (unsigned int), STRING (null-terminated string), MEM (memory range), MAP (mapped memory range), and LABEL (as name server).

For U32, STRING and MEM, when published, DS stores a copy of the data, and programs can retrieve this copy later. U32 and STRING are short enough that the data can be passed directly in a message, so they are more efficient. A MEM value has to be granted access first.

For MAP, DS maps the memory range to DS's address space, which gives a real-time view of that range in a given process. What's more, snapshots can be made any time.

Entries in the 4 types above are stored under a component-specified identifier, which is a string, called key-name. However, the key-name is not unique because of different types. This means that there may be 2 entries will same key-name, but different types. Therefore components have to use both the key-name and type to retrieve/delete the stored data.

For LABEL, it builds an one-to-one (bijection) mapping between a name and a number to allow it to be used as a name server. Because of the bijection, there's a difference with other types; the LABEL entries are not identified by key-name/type. There won't be 2 entries with same name, or with same number. This assures that you can retrieve the name by its number, and retrieve the number by its name.

U32, STRING and MEM entries can be overwritten by key-name; LABEL can be overwritten by label-name or label-num; while MAP can't be overwritten.

All types of entries can be subscribed, through the key-name (or the string if LABEL type). If the subscribed entry is created, changed or deleted, DS will notify the subscriber. An exception is the MAP type. Since its data changes invisibly to DS, the changes will not cause any notifications.

After getting a notification, the subscriber can check which entry was changed, which will give the key-name and type. Then you can use them to retrieve the changed data. Note that one notification may end up being delivered when multiple entries have changed, so the subscriber must call ds_check in a loop until ENOENT is returned.

API

All function prototype and flags (described in the following section) are defined in include/minix/ds.h.

Brief view

• ds_publish_xxx - publish some data to DS.

• ds_retrieve_xxx - retrieve an entry from DS.

• ds_delete_xxx - delete an entry in DS.

• ds_snapshot_map - make a snapshot of mapped memory range.

• ds_subscribe - subscribe changes.

• ds_check - check which entry changed.

U32 functions

ds_publish_u32

int ds_publish_u32(const char *ds_name, u32_t value, int flags);

Publish an unsigned int. If exists, overwrite it.

PARAMETERS:

• ds_name: component-specified identifier

• value: value of unsigned int

• flags: combined by DSF_PRIV_RETRIEVE, DSF_PRIV_OVERWRITE, DSF_PRIV_SUBSCRIB, and DSF_OVERWRITE.

RETURN VALUE:

• If there is no free slot, return EAGAIN;

• If there is already an unsigned int stored, identified by the same ds_name, return EEXIST;

• Otherwise, return OK.

ds_retrieve_u32

int ds_retrieve_u32(const char *ds_name, u32_t *value);

Retrieve an unsigned int.

PARAMETERS:

• ds_name: component-specified identifier

• value: [OUT] contains the value of u32, after return

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

ds_delete_u32

int ds_delete_u32(const char *ds_name);

Delete an unsigned int.

PARAMETERS:

• ds_name: component-specified identifier

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

STRING functions

ds_publish_str

int ds_publish_str(const char *ds_name, const char *string, int flags);

Publish a string.

PARAMETERS:

• ds_name: component-specified identifier

• string: null-terminated string

• flags: combined by DSF_PRIV_RETRIEVE, DSF_PRIV_OVERWRITE, DSF_PRIV_SUBSCRIB, and DSF_OVERWRITE.

RETURN VALUE:

• If there is no free slot, return EAGAIN;

• If there is already a string stored, identified by the same ds_name, return EEXIST;

• Otherwise, return OK.

ds_retrieve_str

int ds_retrieve_str(const char *ds_name, char *value, size_t length);

Retrieve a string.

PARAMETERS:

• ds_name: component-specified identifier

• value: contains the string after return, including null byte

• length: maximum length of the resulting string, excluding null byte

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

WARNING: the supplied buffer must be at least one byte larger than the given length, for the final null byte

ds_delete_str

int ds_delete_str(const char *ds_name);

Delete a string.

PARAMETERS:

• ds_name: component-specified identifier

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

MEM functions

ds_publish_mem

int ds_publish_mem(const char *ds_name, void *vaddr, size_t length, int flags);

Publish a memory range. If specified, it can overwrite an older entry with the new memory range, regardless of its length.

PARAMETERS:

• ds_name: component-specified identifier

• vaddr: virtual address of memory range

• length: length of memory range

• flags: combined by DSF_PRIV_RETRIEVE, DSF_PRIV_OVERWRITE, DSF_PRIV_SUBSCRIB, and DSF_OVERWRITE.

RETURN VALUE:

• If there is no free slot, return EAGAIN;
• If there is not enough memory in DS, return ENOMEM;
• Otherwise, return OK.

ds_retrieve_mem

int ds_retrieve_mem(const char *ds_name, char *vaddr, size_t *length);

Retrieve (part of) a memory range.

PARAMETERS:

• ds_name: component-specified identifier

• vaddr: retrieve the memory range into a buffer beginning from this virtual address.

• length: length of the buffer. DS will retrieve #size data, where #size = MIN(length, length of memory range stored in DS). [OUT] After return, length contains the actual copy data length.

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

ds_delete_mem

int ds_delete_mem(const char *ds_name);

Delete a memory range.

PARAMETERS:

• ds_name: component-specified identifier

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

MAP functions

ds_publish_map

int ds_publish_map(const char *ds_name, void *vaddr, size_t length, int flags);

Publish a part of a memory range, using a mapping.

PARAMETERS:

• ds_name: component-specified identifier

• vaddr: virtual address of memory range, must be aligned to 4K.

• length: length of memory range, must be aligned to 4K.

• flags: combined by DSF_PRIV_RETRIEVE, and DSF_PRIV_SNAPSHOT

RETURN VALUE:

• If there is no free slot, return EAGAIN;
• If there is not enough memory in DS, return ENOMEM;

• If vaddr or length is not aligned to 4K, return EINVAL;

• If there is already a mapping memory range identified by the same ds_name, return EEXIST;

• Otherwise, return OK.

ds_snapshot_map

int ds_snapshot_map(const char *ds_name, int *nr_snapshot);

Make a snapshot of a mapped memory range.

PARAMETERS:

• ds_name: component-specified identifier

• nr_snapshot: [OUT] this will contain the index of the snapshot. This should be used to specify which snapshot needs to be retrieved, as nr_snapshot in ds_retrieve_map().

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• If there is not enough memory in DS, return ENOMEM;
• Otherwise, return OK.

ds_retrieve_map

int ds_retrieve_map(const char *ds_name, char *vaddr, size_t *length, int nr_snapshot);

Retrieve (part of) a mapped memory range or snapshot.

PARAMETERS:

• ds_name: component-specified identifier

• vaddr: retrieve the memory range into a buffer beginning from this virtual address.

• length: length of the buffer. DS will retrieves #size data, where #size = MIN(length, length of memory range stored in DS). [OUT] After return, length contains the actual copy data length.

• nr_snapshot: if DSMF_COPY_MAPPED, retrieve the mapped data; if DSMF_MAP_MAPPED, map the mapped data; or, retrieve the #nr_snapshot snapshot

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• If nr_snapshot is out of range, return EINVAL;

• Otherwise, return OK.

ds_delete_map

int ds_delete_map(const char *ds_name);

Delete a mapped memory range, and its snapshots, if any.

PARAMETERS:

• ds_name: component-specified identifier

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

LABEL functions

ds_publish_label

int ds_publish_label(const char *ds_name, endpoint_t value, int flags);

Publish a label, (name, number);

PARAMETERS:

• ds_name: component-specified identifier, label's name

• value: label's endpoint

• flags: combined using DSF_PRIV_RETRIEVE, DSF_PRIV_OVERWRITE, DSF_PRIV_SUBSCRIB, and DSF_OVERWRITE.

RETURN VALUE:

• If there is no free slot, return EAGAIN;

• If there is already a label with the same name or endpoint, which was published by other process, then: 1) if DSF_OVERWRITE is set in flags, but DSF_PRIV_OVERWRITE was given when published, return EPERM; 2) if DSF_OVERWRITE is not set in flags, return EEXIST;

• Otherwise, return OK.

ds_retrieve_label_endpt

int ds_retrieve_label_num(const char *ds_name, endpoint_t *num);

Retrieve a label's endpoint.

PARAMETERS:

• ds_name: component-specified identifier

• num: [OUT] contains the label's endpoint, after return

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

ds_retrieve_label_name

int ds_retrieve_label_name(char *ds_name, endpoint_t num);

Retrieve a label's name.

PARAMETERS:

• ds_name: a buffer of DS_MAX_KEYLEN in size, that will contain the label's name upon return

• num: label's endpoint

RETURN VALUE:

• If there is no label with number num, return ESRCH;

• Otherwise, return OK.

ds_delete_label

int ds_delete_label(const char *ds_name);

Delete a label.

PARAMETERS:

• ds_name: component-specified identifier

RETURN VALUE:

• If there is no entry identified by ds_name, return ESRCH;

• Otherwise, return OK.

SUBSCRIBE functions

ds_subscribe

int ds_subscribe(const char *regex, int flags);

Subscribe to an entry's changes.

PARAMETERS:

• regex: regular expression for the entries that you want to subscribe to.

• flags: combined using: DSF_OVERWRITE, DSF_INITIAL, and one of 4 types, which are DSF_TYPE_U32, DSF_TYPE_STR, DSF_TYPE_MEM and DSF_TYPE_MAP. If none of these types is set, then subscribe all types.

RETURN VALUE:

• If there is no free slot, return EAGAIN;

• If this process has already subscribed to something, and DSF_OVERWRITE is not set in flags, return EEXIST;

• Otherwise, return OK.

ds_check

int ds_check(char *ds_name, int *type);

Check which subscribed memory range has changed. This function will give you which entry has changed, then you can use the returned key and type to call ds_retrieve_xxx to get the changed data.

PARAMETERS:

• ds_name: a buffer of DS_MAX_KEYLEN in size, that will contain the key name upon return

• type: pointer to an integer that will contain the entry type upon return

RETURN VALUE:

• If this process didn't subscribe to anything, return ESRCH;

• If there is no memory range changed among the process' subscriptions, return ENOENT; This case happens if the entry is deleted before ds_check().

• Otherwise, return OK.

Flags

DSF_PRIV_RETRIEVE, DSF_PRIV_OVERWRITE, DSF_PRIV_SUBSCRIB, DSF_PRIV_SNAPSHOT

• Set private. If one of these flags is set, then this entry will not be retrieved, overwritten, subscribed, or snapshot by other processes. Used when publishing data.

DSF_TYPE_U32, DSF_TYPE_STR, DSF_TYPE_MEM, DSF_TYPE_MAP, DSF_TYPE_LABEL

• Types. Used when subscribing to data.

DSF_OVERWRITE

• If there is an entry identified by the same key-name, if set, overwrite it. Used when publishing data.

DSF_INITIAL

• If set, check whether there is any entry matching this subscription instantly. Used when subscribing to data.

Test

There are 2 system processes to test DataStore, dstest and subs. They are located in /usr/src/test/ds.

dstest tests all new APIs of DS, except ds_subscribe() and ds_check(). test_u32, test_str, test_mem, test_label and test_map test U32, STR, MEM, LABEL and MAP type respectively.

Invalid invocations are tested too. If the same erroneous parameters are tested in former type, it will not be tested in a new type. For example, publishing an entry with same label-name, but without DSF_OVERWRITE set is tested in test_u32, so it's not tested in other types again. New type tests only test new features, which belong to the new type only.

subs tests ds_subscribe() and ds_check(). This server subscribes the U32 (identified by 'test_u32') in dstest. When dstest runs, subs catches it.

See the README with test file for more details.

Others

DS dump

Press Shift+F8 to show the data store dump.

Message

The DS request message is mess_2 ::

• typedef struct {
  • int m2i1, m2i2, m2i3; long m2l1, m2l2; char *m2p1; short m2s1;
  } mess_2;

• m2i1 (DS_KEY_GRANT) and m2s1 (DS_KEY_LEN) are used to contain the grant ID and length of the key name, respectively.

• where needed, m2l1 (DS_VAL) and m2l2 (DS_VAL_LEN) are used to contain the grant ID and length for the value, respectively.

• m2i2 (DS_FLAGS) is used to contain flags.

Page Table

For storing memory, we don't know how much memory DS needs. So DS should have its own page table, to allocate memory dynamically.