BBQueue abstract storage

I'd like for bbqueue to be abstract over the storage medium of the BBBuffer.

There are two main components here:

The tracking header (fixed size)
The data buffer (variable-ish size)

Right now both of these items are stored in the current-day BBBuffer, and they always travel together.

Ideally, we'd like to be able to support any of the following kinds of data buffers:

[u8; N] - a const generic "owned" buffer - likely in static storage. This is the embedded use case, but could also be useful for local usage? Though probably not because lifetimes will get complicated if they are not 'static, but this is still supported today, even if not very useful
&'a mut [u8] - A "user provided" buffer. Again, this is probably not very useful when 'a is not 'static, which would require unsafe to acquire in most cases. However I think that unsafe-ty can be deferred to the caller?
Arc<[u8]>, or something like that

I think the Arc case is really just a subset of the &'a mut [u8] case (where 'a is 'static if handled correctly, but this exposes the necessity to think of the tracking header and the data buffer in the same context.

Really, they need to have the same lifetime. It feels like I should have a struct like this:


#![allow(unused)]
fn main() {
struct BBBuffer {
    header: BBHeader,
    storage: BBStorage,
}
}

And then when handing out something that holds both, they need to have the same lifetime.

I guess not the SAME lifetime, but rather any time we create a reference to BBHeader, it must have a lifetime less than or equal to BBStorage. e.g. 'storage: 'header.

However, BBStorage feels like it should be generic over something. At the very least, I want three potential situations:

A statically allocated "owned" buffer. This needs inner mutability to ensure that the buffer can only be taken once, e.g. a Singleton.
A "borrowed" buffer provided by the user, we just rely on lifetimes and the borrow checker to make sure this works out.
A reference counted buffer, which de-allocates itself when no handles (e.g. Buffer, Producer, Consumer, Grants) are held.

Brainstorming with fake code

What if we make BBStorage a trait? This way I could provide some convenience types for the three above situations, and a user could also construct their own if they had some other use case, like shared memory between CPUs or IPC.

I worry a little bit about the generic types making the API look pretty garbage or overly complicated. Especially with fully qualified types when using as a static. I wonder if I can "paper over" this with type aliases or dependant traits.


#![allow(unused)]
fn main() {
// I don't know if this lifetime syntax is right
trait BBStorage<'header, 'storage: 'header> {
    // We need some kind of "associated lifetimes"
    type StorageLifetime = PhantomData<'storage>;
    type HeaderLifetime = PhantomData<'header>;

    // I think `self` needs to have `'storage` lifetime, as it is longer.
    fn get_storage(&'storage self) -> &'storage impl Deref<Target=Storage>;
    fn get_header(&'storage self) -> &'header impl Deref<Target=Header>;
}
}

I think with this approach, I could:

Create a struct that owns both Storage and Header (with a const constructor), which each would have the same lifetime as the storage structure
Create a struct that borrows both Storage and Header
A wrapper type that holds an Arc of the owned version, and returns a reference

But wait, if these two lifetimes are always the same, can't I just have a parent structure that references both children?

The main problem is I need some kind of "hook" to increment refcounts in the Arc case. I wonder if I can simplify it to just this:


#![allow(unused)]
fn main() {
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

trait BBGetter<'bbqueue> {
    fn get_borrow(&'bbqueue self) -> impl Deref<Target = BBBorrow<'bbqueue, 'bbqueue>;
}
}

Hmm, maybe? Not sure. I think the issue here is that all the types need to be generic over the BBGetter. This would mean that I need:


#![allow(unused)]
fn main() {
struct Producer<B: BBGetter> {
    bbq: B,
    // ...
}

struct GrantW<B: BBGetter> {
    bbq: B,
    // ...
}
}

I'm dancing around the concept of BBBorrows being Clone-able, but those clones would be bound to the lifetime of the BBBorrow itself, which is what I want, and in the case of 'static, this is no problem. I just need the borrow checker to understand that BBBorrow can't outlive 'storage, which I think the borrow checker can understand already?

Okay, let's go write some playground code to prove this out.

Attempt 1

use core::sync::atomic::AtomicUsize;
use std::sync::Arc;

struct Header {
    fake: AtomicUsize,
}

struct Storage {
    fake: AtomicUsize,
}

struct OwnedBuffer {
    hdr: Header,
    sto: Storage,
}

#[derive(Clone)]
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

struct BBArc {
    barc: Arc<OwnedBuffer>,
}

trait BBGetter<'hdr, 'sto: 'hdr> {
    fn get_borrow(&self) -> BBBorrow<'hdr, 'sto>;
}

impl<'hdr, 'sto: 'hdr> BBGetter<'hdr, 'sto> for BBBorrow<'hdr, 'sto> {
    fn get_borrow(&self) -> BBBorrow<'hdr, 'sto> {
        return self.clone()
    }
}

// impl BBGetter<'static, 'static> for BBArc {
//     fn get_borrow(&self) -> &BBBorrow<'static, 'static> {
//         return self.barc.get_borrow()
//     }
// }


fn main() {
    println!("Hello, world!");
}

Attempt 2

use core::sync::atomic::AtomicUsize;
use std::sync::Arc;

struct Header {
    fake: AtomicUsize,
}

struct Storage {
    fake: AtomicUsize,
}

struct OwnedBuffer {
    header: Header,
    storage: Storage,
}

impl OwnedBuffer {
    fn borrow(&self) -> BBBorrow {
        BBBorrow {
            header: &self.header,
            storage: &self.storage,
        }
    }
}

#[derive(Clone)]
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

#[derive(Clone)]
struct BBArc {
    barc: Arc<OwnedBuffer>,
}

trait BBGetter<'hdr, 'sto: 'hdr>: Clone {
    fn get_header(&self) -> &'hdr Header;
    fn get_storage(&self) -> &'sto Storage;
}

impl<'hdr, 'sto: 'hdr> BBGetter<'hdr, 'sto> for BBBorrow<'hdr, 'sto> {
    fn get_header(&self) -> &'hdr Header {
        &self.header
    }
    fn get_storage(&self) -> &'sto Storage {
        &self.storage
    }
}

impl<'hdr, 'sto: 'hdr> BBGetter<'hdr, 'sto> for BBArc {
    fn get_header(&self) -> &'hdr Header {
        // Can't borrowck
        self.barc.borrow().get_header()
    }
    fn get_storage(&self) -> &'sto Storage {
        todo!()
    }
}


fn main() {
    println!("Hello, world!");
}

Attempt 3

The problem with this attempt is that the Producer and Consumer need to OWN something that can reference the data, but NOT for longer than the reference to the Arc (in this case) lives. How can I introduce this?

use core::sync::atomic::AtomicUsize;
use std::sync::Arc;
use core::marker::PhantomData;

struct Header {
    fake: AtomicUsize,
}

struct Storage {
    fake: AtomicUsize,
}

// TODO: Wrap this in a "Singleton"?
// Should this be a private type? This probably
// wont play nice with being `'static`...
// Probably requires `MaybeUninit` and such
struct OwnedBuffer {
    header: Header,
    storage: Storage,
}

impl OwnedBuffer {
    // Should this be a private or unsafe method?
    fn borrow(&self) -> BBBorrow {
        BBBorrow {
            header: &self.header,
            storage: &self.storage,
        }
    }

    fn split(&self) -> Result<(Producer<BBBorrow>, Consumer<BBBorrow>), ()> {
        Ok((
            Producer {
                bbq: self.borrow(),
                pds: PhantomData,
                pdh: PhantomData,
            },
            Consumer {
                bbq: self.borrow(),
                pds: PhantomData,
                pdh: PhantomData,
            }
        ))
    }
}

#[derive(Clone)]
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

#[derive(Clone)]
struct BBArc {
    barc: Arc<OwnedBuffer>,
}

trait HeaderGetter<'hdr>: Clone {
    fn get_header(&self) -> &'hdr Header;
}

trait StorageGetter<'sto>: Clone {
    fn get_storage(&self) -> &'sto Storage;
}

impl<'hdr, 'sto: 'hdr> HeaderGetter<'hdr> for BBBorrow<'hdr, 'sto> {
    fn get_header(&self) -> &'hdr Header {
        &self.header
    }
}

impl<'hdr, 'sto: 'hdr> StorageGetter<'hdr> for BBBorrow<'hdr, 'sto> {
    fn get_storage(&self) -> &'sto Storage {
        &self.storage
    }
}

impl<'hdr, 'sto: 'hdr> HeaderGetter<'hdr> for BBArc {
    fn get_header(&self) -> &'hdr Header {
        &self.barc.header
    }
}

impl<'hdr, 'sto: 'hdr> StorageGetter<'hdr> for BBArc {
    fn get_storage(&self) -> &'sto Storage {
        &self.barc.storage
    }
}

struct Producer<'hdr, 'sto: 'hdr, G>
    where G: StorageGetter<'sto> + HeaderGetter<'hdr>,
{
    bbq: G,
    pds: PhantomData<&'sto ()>,
    pdh: PhantomData<&'hdr ()>,
}

struct Consumer<'hdr, 'sto: 'hdr, G>
    where G: StorageGetter<'sto> + HeaderGetter<'hdr>,
{
    bbq: G,
    pds: PhantomData<&'sto ()>,
    pdh: PhantomData<&'hdr ()>,
}



fn main() {
    println!("Hello, world!");
}

Attempt 4

Hmm, this feels better, though it feels like .borrow() and .split() could be part of a combined Getter trait that doesn't split the header and storage borrows...

use core::sync::atomic::AtomicUsize;
use std::sync::Arc;
use core::marker::PhantomData;

struct Header {
    fake: AtomicUsize,
}

struct Storage {
    // TODO: This should probably be something
    // like `MaybeUninit<[u8; N]>`
    fake: AtomicUsize,
}

// TODO: Wrap this in a "Singleton"?
// Should this be a private type? This probably
// wont play nice with being `'static`...
// Probably requires `MaybeUninit` and such
struct OwnedBuffer {
    header: Header,
    storage: Storage,
}

impl OwnedBuffer {
    // Should this be a private or unsafe method?
    fn borrow(&self) -> BBBorrow {
        BBBorrow {
            header: &self.header,
            storage: &self.storage,
        }
    }

    fn split(&self) -> Result<(Producer<BBBorrow>, Consumer<BBBorrow>), ()> {
        Ok((
            Producer {
                bbq: self.borrow(),
            },
            Consumer {
                bbq: self.borrow(),
            }
        ))
    }
}

impl BBArc {
    // Should this be a private or unsafe method?
    fn borrow(&self) -> BBBorrow {
        self.barc.borrow()
    }

    fn split(&self) -> Result<(Producer<BBArc>, Consumer<BBArc>), ()> {
        Ok((
            Producer {
                // bbq: self.borrow(),
                bbq: self.clone()
            },
            Consumer {
                // bbq: self.borrow(),
                bbq: self.clone()
            }
        ))
    }
}

#[derive(Clone)]
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

#[derive(Clone)]
struct BBArc {
    barc: Arc<OwnedBuffer>,
}

trait HeaderGetter: Clone {
    fn get_header(&self) -> &Header;
}

trait StorageGetter: Clone {
    fn get_storage(&self) -> &Storage;
}

impl<'hdr, 'sto: 'hdr> HeaderGetter for BBBorrow<'hdr, 'sto> {
    fn get_header(&self) -> &Header {
        &self.header
    }
}

impl<'hdr, 'sto: 'hdr> StorageGetter for BBBorrow<'hdr, 'sto> {
    fn get_storage(&self) -> &Storage {
        &self.storage
    }
}

impl HeaderGetter for BBArc {
    fn get_header(&self) -> &Header {
        &self.barc.header
    }
}

impl StorageGetter for BBArc {
    fn get_storage(&self) -> &Storage {
        &self.barc.storage
    }
}

struct Producer<G>
    where G: StorageGetter + HeaderGetter,
{
    bbq: G,
}

struct Consumer<G>
    where G: StorageGetter + HeaderGetter,
{
    bbq: G,
}



fn main() {
    println!("Hello, world!");
}

Let's try combining the traits and see what that does to the lifetimes everywhere...

Attempt 5

Okay, combining the traits works, but I don't think I can move split to the trait, because that NEEDS to be a singleton.

The Arc option could never give you the original BBArc back for convenience, leaving split to be a static singleton artifact.

use core::sync::atomic::AtomicUsize;
use std::sync::Arc;
use core::marker::PhantomData;

struct Header {
    fake: AtomicUsize,
}

struct Storage {
    // TODO: This should probably be something
    // like `MaybeUninit<[u8; N]>`
    fake: AtomicUsize,
}

// TODO: Wrap this in a "Singleton"?
// Should this be a private type? This probably
// wont play nice with being `'static`...
// Probably requires `MaybeUninit` and such
struct OwnedBuffer {
    header: Header,
    storage: Storage,
}

impl OwnedBuffer {
    // Should this be a private or unsafe method?
    fn borrow(&self) -> BBBorrow {
        BBBorrow {
            header: &self.header,
            storage: &self.storage,
        }
    }

    fn split(&self) -> Result<(Producer<BBBorrow>, Consumer<BBBorrow>), ()> {
        Ok((
            Producer {
                bbq: self.borrow(),
            },
            Consumer {
                bbq: self.borrow(),
            }
        ))
    }
}

impl BBArc {
    // Should this be a private or unsafe method?
    fn borrow(&self) -> BBBorrow {
        self.barc.borrow()
    }

    fn split(&self) -> Result<(Producer<BBArc>, Consumer<BBArc>), ()> {
        Ok((
            Producer {
                // bbq: self.borrow(),
                bbq: self.clone()
            },
            Consumer {
                // bbq: self.borrow(),
                bbq: self.clone()
            }
        ))
    }
}

#[derive(Clone)]
struct BBBorrow<'header, 'storage: 'header> {
    header: &'header Header,
    storage: &'storage Storage,
}

#[derive(Clone)]
struct BBArc {
    barc: Arc<OwnedBuffer>,
}

trait BBGetter: Clone {
    fn get_header(&self) -> &Header;
    fn get_storage(&self) -> &Storage;
}

impl<'hdr, 'sto: 'hdr> BBGetter for BBBorrow<'hdr, 'sto> {
    fn get_header(&self) -> &Header {
        &self.header
    }

    fn get_storage(&self) -> &Storage {
        &self.storage
    }
}

impl BBGetter for BBArc {
    fn get_header(&self) -> &Header {
        &self.barc.header
    }

    fn get_storage(&self) -> &Storage {
        &self.barc.storage
    }
}

struct Producer<G>
    where G: BBGetter,
{
    bbq: G,
}

struct Consumer<G>
    where G: BBGetter,
{
    bbq: G,
}



fn main() {
    println!("Hello, world!");
}

Thoughts on 6

I should have three types exposed:
- ArcBBQueue
  - Arc
  - Only have an interface create prod/cons
    - Takes ownership
- OnceBBQueue
  - Singleton
  - Interface to create prod/const
    - Takes &self
- BorrowBBQueue // Never instantiatable
  - Wait, how can we "take ownership" of an immutable borrow?
  - I guess we just have something like:
    - unsafe fn with_storage(&'sto mut [u8]) -> Borrow<'sto>
  - Can this ever be sound?

James' Lab Notebook