Edge Queues
Edge Queues
Design
- Belongs to a space (identified as space id)
- RB: Could have multiple per space; queue ID separate; parentID is space.
- Used for sharding
- Different kind of space then the usual echo space
- multiple queues per space
- Each queue is identified by its ID
- Same schema as ObjectID
- Queue has a DXN of
dxn:queue:<spaceId>:<queueId>
- Queues are automatically created when the first object is inserted
- Queues contain typed ECHO objects
- Each object has an echo id
- RB: Effectively yes, but strictly just a typed object (schema).
- Each object in a queue can be identified with
dxn:queue:<spaceId>:<queueId>:<objectId>
- Each has a typename (with version and schema reference)
- Single queue can contain objects of mixed types
- Each object has an echo id
- Queue orders the objects in the insertion order
- Consumers can query the front/tail of the queue
- Queues expose opaque cursors that can be used for ranged queries - client maintains cursor.
RB: I think this can be simplified to a range query (client maintains cursor).
- Queues expose opaque cursors that can be used for ranged queries - client maintains cursor.
- Consumers can delete objects from a queue
- Using cursors
- Using ECHO ids
- Consumers can query the front/tail of the queue
- Objects inside a queue can be garbage-collected
- Queues can be configured with expiration time for their items.
- RB: Ideally messages also (note: CF KV store has automatic expiration for records: could use to track set of queues?)
- Queues can be configured with expiration time for their items.
- Objects in the queue are mutable
- We use lamport timestamps for versioning.
- RB: Might be simpler to make it just immutable and track monotonic versions. I.e., message itself is immutable.
- We use lamport timestamps for versioning.
Implementation
- One durable object per queues space
- RB: Meaning 1 DO for all queues for a given space? Why do queses need a DO?
- Using sqlite for DOs as this seems to be the direction cloudflare is moving towards
- Schema
queue-entries
CREATE TABLE queue_entries ( -- auto-incremented key used for range queries id INTEGER AUTOINCREMENT PRIMARY KEY, version TEXT, -- Reserved for the lamport clock of the object version, NULL for now object_id TEXT, -- ECHO object id of each object queue_id TEXT, type_dxn TEXT, -- Extracted from echo object for faster lookup -- [RB]: schema_dxn? created_ts INTEGER, -- Unix timestap of the creation time updated_ts INTEGER, -- Unix timestap of the update time data TEXT, -- Object data in JSON format INDEX idx_range (queue_id, id), -- For efficient range queries. - [RB]: Do we need an index for TS range queries? UNIQUE INDEX idx_object_id (queue_id, object_id), -- Ensures that queue don't have duplicate object ids. - [RB]: Is the DXN of an object: space-queue-object? );
- NotesRB
- Retrofit to AI service.