Stores Actors Data Tree Downloads
AnalysisObject size

Object size mature

Objects are the smallest piece of data in the Condensation model. Although some protocols allow partial downloads (e.g., HTTP range requests), the integrity of an object can can only be checked if the whole object is available. Objects are simply not meant to be divided further.

In practice, objects are often between 1 KiB to 100 MiB in size. Larger content is split into several objects, and stored as object tree.

The optimal object size depends on how the data is accessed. In general, objects should be smaller the more often the underlying data is modified. In addition, data should be split into several objects if only small parts are needed at a time. This avoids downloading a large object to extract a small piece of information.

On the other hand, each object incurs a storage overhead of about 100 bytes, and a similar transmission overhead.

Size limits

The size of an object is not inherently limited. While the hash list may contain at most 2^32 hashes, the data section may be arbitrarily long.

Object stores however impose practical limits, often somewhere between 10 GiB and 1 TiB. The maximum object size is limited either by the remaining space, or some parameter intrinsic to the storage system. Some stores also keep objects fully in memory during the transfer, and may therefore be limited by the available memory. Object stores may even refuse exceedingly large objects.

It is safe to assume that an object store can handle objects up to 1 GiB in size.

Obtaining the size of the object

The Condensation store protocol does not contain any standard function to obtain the size of an object. The result of such a function would not be verifiable by the client, limiting its usefulness.

To measure the size of an object, a client must download the object (and verify its integrity).

Applications may store the size of relevant objects or subtrees (e.g. photo collection) as part of their private data. Since objects and subtrees are immutable, this is straightforward.