#12 – James Pearce: Tinybase

July 23, 2024

The guest of this episode is James Pearce, the author of Tinybase, a reactive data store library for local-first apps. This conversation will explore how Tinybase works including its custom query system, the various persistence and syncing integrations as well as James’ plans for the future.

Mentioned in podcast

James Pearce: x.com/jamespearce + scoutsailing.com
TinyBase
TinyHub

Links:

Thank you to Expo and Rocicorp for supporting the podcast.

Transcript

#12 – James Pearce: Tinybase

00:00Well, can I just say that, you know, from the point of view of exhaustiveness,

00:04there are many, many unsolved problems in the local-first space.

00:09And I have a horrible feeling that it's easy to think that

00:13synchronization is the problem.

00:14And then once you've solved synchronization, you're good.

00:17Well, I don't think that's true.

00:20I think we all got hung up on, oh, once we crack CRDTs, we'll be good.

00:23no, no, no, no, that's the easy part.

00:25It's not state management, it's the state of the state management in a way.

00:28It's like, should I be synchronizing this data?

00:31Because this person isn't logged in, or is logged in, or is online, or is

00:35offline, or has shared this document, or hasn't shared this document, and

00:39that is actually unsolved, I think.

00:42Welcome to the localfirst.fm podcast.

00:44I'm your host, Johannes Schickling, and I'm a web developer, a

00:47startup founder, and love the craft of software engineering.

00:51For the past few years, I've been on a journey to build a modern, high quality

00:55music app using web technologies.

00:57And in doing so, I've been falling down the rabbit hole of local-first software.

01:01This podcast is your invitation to join me on that journey.

01:05In this episode, I'm speaking to James Pearce, the author of TinyBase, a reactive

01:09datastore library for local-first apps.

01:12In this conversation, we explore how TinyBase works, including its

01:16custom query system, the various persistence and syncing integrations,

01:20as well as James plans for the future.

01:22Before getting started, also a big thank you to Rosicorp and

01:25Expo for supporting this podcast.

01:28And now my interview with James.

01:31Hey, welcome James.

01:32Nice to meet you and nice to have you on the show.

01:34How are you doing?

01:35I'm doing great.

01:36Thank you so much for having me.

01:37Huge privilege to have a chance to come along and chat

01:40about all things local-first.

01:42yeah, very excited to be here.

01:43It's safe to say that you're the first guest so far in the podcast,

01:47who's calling in from a boat, which I think is very on brand for what

01:52local-first software should enable.

01:55and I think Maybe there's a sort of romantic pitch for why you're doing

01:59local-first, or, or maybe not so much.

02:02I think we'll get to that in a moment, but the first time, I've took notice of

02:07you is like when you've been at Facebook back then taking care of overseeing a

02:12whole bunch of open source projects.

02:13So would you mind taking a step back, briefly introducing yourself and sharing

02:18more about, your time at Facebook?

02:20Sure.

02:21Certainly.

02:21So, I'm James.

02:22I did have a career before Facebook slash Meta, fairly long time ago.

02:27I'm from the UK originally.

02:28I've been in tech nearly all of my professional life.

02:31Moved to California in about, 2010, and started work at Facebook and worked on a

02:38variety of things during my time there.

02:40I was at the company for just over 10 years, but, you know, one of the

02:44highlights for sure was working on.

02:46The open source program, as I'm sure most of the listeners are aware Meta

02:51does have quite a large investment in open source whether it's on the client

02:55JavaScript technology side or on DevTools or on data infrastructure and of course

03:02now on AI, ML and LLM technologies.

03:07And so yeah, it was a, an awesome period of time for me.

03:10It was between about 2014 and 2017.

03:14So I helped out with a bunch of the launches of things like React

03:18and PyTorch and HHVM and various build tools and a whole bunch of things

03:24that people are probably aware of.

03:26And yeah, it was a super exciting time, just in terms of helping to develop how

03:31the company thought about open source and what open source could do for the company.

03:36Wasn't the only thing I did whilst I was at Facebook.

03:38Also worked on Portal, which was the smart speaker device that the company had.

03:43, sadly now discontinued, but, it was, that was pretty awesome working on,

03:46on a hardware project for a bit.

03:48And also spent some time in the data infrastructure team, working on some of

03:52the large data warehouse technologies, that also Meta is fairly well known for.

03:57So, yeah, one of the great things, maybe underappreciated, features of

04:01being an employee at Facebook is you, you do get to, move around a lot.

04:04There are lots of mobility opportunities, and I took the greatest advantage of

04:08that, that I could, and worked on a wide, wide range of different things.

04:11but yeah, enjoyed, every minute of it and then in about 2022.

04:16So yeah, just coming up on two years ago, figured it was a, a good opportunity

04:20to try something completely different.

04:21And so, Moved on, moved on board Scout, which is the boat that you see me on now.

04:27And, sold up, my house and life in Silicon Valley.

04:31And I've been sailing on the ocean waves ever since writing open source

04:35software, so that's where we are.

04:36That sounds quite incredible.

04:38And I mean, Facebook is certainly.

04:41One of the companies that comes to mind is like how to have a positive sum

04:46vision on how to do open source at scale.

04:49, I can't really think of another company that has had such an, such a

04:53positive impact on our web ecosystem.

04:56and in particular With React, with GraphQL.

05:00I'm not as deep on the AI side, but I mean, the recent releases,

05:05et cetera, around Llama, et cetera, this is all, I think a great example

05:09of how a large company can have a very positive impact on the, the

05:15ecosystem beyond its own benefits.

05:17So it's great to hear that you must've had like a very positive impact on that.

05:22Yeah.

05:22It's a lot of this.

05:23took time to develop.

05:24I mean you could argue we stumbled into some of these benefits inadvertently.

05:29I think early on it just seemed like the right thing to do.

05:32The company had been built on other people's open source when, you know,

05:36Mark Zuckerberg started the site.

05:38PHP, running on Apache with MySQL.

05:41And so there was always this sense of standing on the shoulders of other giants.

05:44And so there was a, an almost cultural obligation to share back things like

05:49Memcached and, you know, some of the early things that the company was working on.

05:53but then it began to become apparent that it was more than just that obligation.

05:57It was also, starting to help lead the industry because as Facebook

06:01grew, it was experiencing problems that other apps or other sites

06:06weren't, just because of its scale.

06:07And so it was like the advanced guard of finding out what happens when you

06:11build really complicated web apps.

06:13and so that was what gave birth to React, you know, at the

06:16time that React was launched.

06:18You know, most web apps were still MVC that was kind of the prevailing

06:22pattern and it was still, still super easy to have inconsistencies between

06:27state across a big complicated app.

06:29And so, you know, React's innovation was, you know, how do we make sure that

06:32the, the user interface is always a pure function of data that's gone into it.

06:37And, you know, at the time it seemed heretical, but pretty quickly other

06:40people started to hit the same.

06:42scaling issues with, with MVC apps and, and, and realized

06:45that React was maybe a way out.

06:46And that's the pattern I think at Facebook over and over again, whether it's data

06:50infrastructure or or even ML, you know, the, the company is experiencing problems

06:55that others will face, but haven't yet.

06:57And so it has a chance to maybe share back some of its solutions from the future, so

07:02to speak, so that others then benefit from those as they experience those challenges.

07:06Um, so that was, that was one of the big drivers, I think.

07:09Obviously, it helps with recruiting, right?

07:10Because you're demonstrating the kinds of problems that you're solving

07:13and that gets people excited to come and help solve those problems.

07:16and it also kind of opens the windows onto the kind of code that people are

07:20building at a company like Facebook.

07:22And obviously, there are lots of world class engineers there, and there's no

07:26better way of learning that than seeing the kind of code that they're writing.

07:29, and so it kind of gives you that little bit of visibility into the, the caliber of

07:34the, the engineers that are working there.

07:35And yeah, so there, there were benefits coming back to the company

07:39and just in terms of engineering brand and recruiting, that over time became,

07:43that became so obvious that it was like, okay, now we need to just keep

07:45investing in all of these things.

07:47Um, and it's now more than just a thing that we're doing for fun.

07:50It's now something that's actually bringing real value to the business.

07:54Exactly.

07:54And I think there's no coincidence that now some of the most prolific, open

08:01source and tool builders, actually used to work at Facebook, like thinking of

08:05the people who invented GraphQL, now building other technologies or Matt

08:10Wonlaw, who was just on the show, was also at Facebook for, for a while.

08:15So I think really like this deep culture.

08:17around great developer experience and very principled thinking,

08:21like marrying those two together.

08:22I think this is what, what makes tools really successful and sticky.

08:27And I think this is what the internal, Facebook engineering culture has

08:32sort of like brought to light.

08:34I will say it's not as easy as it sounds.

08:36there are lots of things you have to do behind the scenes and as

08:38someone that was responsible.

08:40for the the overall program rather than individual projects, you know, I

08:44needed to think about how we I guess sell this concept internally, right?

08:49Because if you're an engineering manager and you're running a team and they've

08:53decided they want to spend 20 percent of their time supporting their project on

08:57github You're like wait, but now i'm down 20 percent in terms of like my internal

09:00goals So a lot of it was a lot of the challenges were internally Making sure

09:05that everybody was on board with the, the macro global benefits, even if at a

09:09local level, it seemed like it was a cost.

09:12but we got around that in a variety of different ways.

09:14, one of the ways that, you know, perhaps isn't obvious, but I will

09:16state it because I think it was, it was perhaps even innovative at the

09:20time, is make sure you're using exactly the same code from GitHub internally

09:25and I see lots of companies that, you know, are doing open source, but

09:28it's a fork that they've thrown over the wall from a couple of years ago.

09:31And now they've diverged and they're doing something completely different internally.

09:34and that never works because of course that always gets forgotten about relative

09:38to the thing that's being used internally.

09:39But I think it's pretty awesome that you go to GitHub and you look at slash

09:44react, and then you go to facebook.

09:46com and you view source.

09:48And you will see exactly the same code, because the, the internal version of

09:52React and the external version are being synchronized commit by commit.

09:56and so that means that you never get out of sync.

09:58It means that the engineers who are working internally get to

10:01see their value immediately going to the community and vice versa.

10:04The community gets to see their value, appearing on the world's largest website.

10:07So, those things are hard, like, infrastructurally to get working, but

10:11those are the kinds of things that, that then help the whole open source machine

10:15work inside a large company like that.

10:18So in 2022, you mentioned you sold your house, you, like, made a drastic change to

10:24your life, and you moved onto your boat.

10:27how Did that lead you to local-first?

10:29It seems to coincide a bit timing wise to, to also, start like working

10:35on TinyBase as your primary project.

10:37Can you share more about that?

10:39Yeah, I guess with the benefit of hindsight, you can rework the

10:42origin story a little bit and say, Oh yeah, it was all intentional.

10:45it wasn't, of course.

10:46I had.

10:47Some good milestones in my life you know, my, my kids were leaving home and

10:52had reached my 10 year anniversary at Facebook and the boat that we had been

10:57dreaming about, you know, was, was ready.

10:59and so it seemed like it was a good idea to do all these, these things at the same

11:02time in terms of my, my life choices.

11:05but as regards local-first and TinyBase, which is the project that I've been

11:10working on that I'm sure we'll, we'll talk about, that was somewhat Coincidental,

11:15that was something that I had been, you know, thinking about for a long time.

11:18but I think, yeah, then once you move on a boat, the, a number of the

11:23benefits of, of building local-first apps suddenly become acutely clear.

11:27It's like, wait, Oh, sometimes I really just don't have a cloud.

11:30And sometimes even if I do, my network is pretty bad.

11:34And it really throws into perspective the difference between the apps that

11:38I native apps, right, that do have much better local support and the web

11:44apps, which just now fail to load.

11:46And it really threw into stark relief the difference between them at the

11:50same time, as I said, I was working on TinyBase and maybe I can reverse the local

11:55local-first principle into into the origin story of TinyBase because it does seem

11:59to help support those kinds of things.

12:01and then like almost within a month, I then read the Ink and Switch article

12:05and I saw the essay about Rffle.

12:07I'm like, okay, all my, all these things are converging and , and

12:12it all suddenly made sense.

12:13So there was a lot of serendipity involved in kind of weaving

12:17these narratives together.

12:18But yeah, certainly life on a boat does, does make you think about

12:23local-first a lot more than living in some, you know, high density,

12:26high network connectivity lifestyle.

12:28So, yeah, no, it definitely helped inform my thinking about this and has encouraged

12:33me to, to keep working on, on TinyBase and other, other projects like it.

12:36So you mentioned keep working on TinyBase.

12:38So something must've led you to start working on TinyBase to begin

12:43with, and later you oriented more around local-first, what was the

12:48impetus to start working on TinyBase?

12:50So, well, maybe I should just like rewind a little bit in case

12:54it's not blindingly obvious.

12:56I love building stuff and all of my life, I guess, I've been, you

13:01know, surrounded by computers.

13:02I was programming when I was a kid and, you know, grew up always trying to

13:06create things with code and I guess the lifestyle of a manager or a director at

13:12Meta is that you tend to get a little further and further away from the code

13:17base, which I found sad because even though I love managing teams and people

13:22and helping grow teams, Organizations and so forth, like some part of me,

13:27like a primeval part of me still needs to be building something of my own.

13:31And so, that, that was really the origin of just doing something, anything.

13:36I just needed to be able to keep building even if it was

13:39just a small project for myself.

13:41And then obviously once I left the company and I started to find I had

13:44more time, I was able to start investing a little bit more effort into it.

13:49And yeah, the, the original kind of story for, for building TinyBase was that I

13:55had an idea in my mind of an app that would be really cool, but I knew that

13:59in order to build that app, I was going to need to have something like TinyBase.

14:03So, you know, without going into too much detail, you know, when we were

14:06at when I was at Facebook, I, I, I needed a way to manage the state of

14:11hundreds of open source projects, right?

14:12And so we built some internal tools that allowed us to, Pull from GitHub, you

14:18know, the state of all of the repos, how many pull requests, issues, and so forth.

14:22And um, That tooling really made it easy for us to look at this

14:26huge portfolio and figure out what was going well and what wasn't.

14:29But I felt like the world itself could benefit from something like

14:33that So I thought maybe maybe there's a there's an app out there somewhere

14:36which would be , a GitHub at scale kind of interface that would let you

14:40manage your open source portfolio.

14:42And so anyway, I thought, well, if I needed an app like that, and I'd need

14:45to put it in a browser, and I'd need to have some kind of way of storing

14:49the tables of repos and issues and pull requests, you know, how would I do that?

14:53Like, what is a good tabular data store that runs in the browser?

14:57And there isn't, well, back then anyway, 2021 2022, there wasn't

15:02really an obvious way to do that.

15:04And so I thought, well, rather than building the app, I'll start off by

15:07building a state management library that allows me to store tabular data that in

15:13the future will store this data that I need in this app that I haven't built yet.

15:16And yeah, that, that, that's where TinyBase originally came from.

15:20I knew it needed sort of database semantics, but didn't actually

15:24need to be a database, right?

15:25It could still be considered an in memory data store.

15:28and I'm sure everyone's familiar with the idea of a, you know,

15:31kind of a store in a React app.

15:33But I wanted it to kind of look and feel a little bit like a database.

15:36I wanted it to feel like it had tables and rows and cells.

15:39I wanted to do queries against it, and I wanted to be able to do aggregates

15:42of the data so I could count up the number of stars or commits in a

15:45repo and all those kinds of things.

15:47so yeah, that was the original, idea of TinyBase, you know, something

15:51that wasn't a database, but sort of had database conceptual semantics.

15:56and that's where it popped up.

15:57And then of course I stumbled into the Rffle And you

16:01know, Ink and Switch material.

16:03And I obviously spotted that you were doing something similar with,

16:06with a SQLite approach, which is obviously literally a database.

16:09So you literally get the database semantics.

16:12but yeah, by that point I'd kind of committed myself to

16:14doing an in memory approach.

16:17So I thought, well, let's, let's try both.

16:20, and I, and so I kept pushing, pushing forward with TinyBase.

16:23Now the story has obviously evolved a lot since then.

16:25TinyBase is now a lot better about integrating with real databases.

16:29We can come on to talk about that.

16:30Um, but yeah, that was, that was the origin story.

16:33It was just something I needed for myself.

16:35I thought the world needed, and I went and built it.

16:37And then once I found I had more time, I guess I kept building.

16:41that is amazing.

16:41I think there's a lot of similarities in terms of your motivations, et cetera.

16:46Maybe a few different technical decisions along the way, but for me,

16:51starting to work on Overtone would have not been feasible without having

16:56a very substantial State, local state management foundation to build on top of.

17:01And this is how I then like got to know Jeffrey, et cetera, and

17:05started getting involved with Rffle.

17:07So it seems like there is a clear parallel to you wanting to build your own app

17:12and realizing, okay, that's going to be terrible working on that app without

17:17those primitives that you have in mind.

17:19So obviously you got to start building your, your own state

17:22management system first.

17:24so, and kudos to you.

17:26I think TinyBase is one of the most polished and, sophisticated solutions

17:31that is already out there where I think a lot of technologies are, maybe have

17:37really great ideas, but are still.

17:40pretty early in terms of being approachable and having the

17:43supporting resources around it.

17:45So, for example, LiveStore is not yet open source.

17:48It's available to GitHub sponsors, but it doesn't have documentation yet.

17:52It doesn't have guides yet.

17:53It doesn't have, like, all of those, supporting material around it.

17:57Whereas TinyBase has a gorgeous website.

18:00Really polished docs, lots of examples, lots of guides.

18:05you keep, releasing regularly, like new versions, integrations

18:09with different other technologies.

18:11So really, really exciting how far you've come.

18:14I'm now looking forward to trying to understand better the different details.

18:18Right.

18:19Well, just firstly, a bit, a little bit of a meta point on

18:22the, on the kind of the polish.

18:23so a side kind of motivation for this was to see what it was like

18:28to build software where I didn't have to make any major compromises.

18:33and I'm sure you and, and, and the people listening or watching, you

18:36know, have, have written software where you're under a deadline or there's

18:40some set of constraints and you just start cutting corners, you know, ah,

18:44you know, this bit of code is really crappy or, well, we didn't have time

18:47to finish the test coverage or, no, just, we've got to ship it, right?

18:50Cause that's, and actually that's fine.

18:52Right.

18:52I mean that, that, that's life.

18:56I always feel like some.

18:57Sort of perfectionist part of my brain is always like, ah, like next time,

19:02next time, I'm going to make it perfect.

19:03Next time, you know, the architecture is going to be perfect.

19:06Next time the test coverage is going to be a hundred percent.

19:09And so given that I now had this project that only I was working on,

19:13I was really only my only customer.

19:15I had a bunch of spare time.

19:17I figured.

19:18, if I'm ever going to have a chance to build, quote, what I thought was

19:21perfect software, and of course that's subjective, it's just my view that it's

19:23perfect then this was going to be it.

19:25So, can I build a piece of software that has, you know, 100 percent test coverage,

19:30is refactored, like, so iteratively that eventually it can't be refactored anymore.

19:36, it's got all the documentation you could ever need.

19:39It's got the best website that me and my crappy web design can manage.

19:43Um, and I, you know, I don't have a deadline, right?

19:47I don't have somebody breathing down my neck.

19:48I don't have some investors I got to keep happy.

19:51It's like, no, it's like, I'm going to polish this thing.

19:54It's, and it's a beautiful, feeling of kind of craftsmanship, which I have

19:59rarely had before in my professional life.

20:01You know, often you feel like you're a tradesman.

20:03You're just like knocking stuff out for a customer or for a deadline.

20:06But suddenly I could feel like, sounds pretentious, but

20:09almost like an artist, right?

20:10You know, artisanal software that's like perfectly polished

20:14to my, to my specification.

20:16And that was an interesting journey in its own right.

20:18By the way, it's also a curse because you could just end up polishing and

20:22polishing and polishing and then forgetting to actually share it with

20:25you, share it with the rest of the world.

20:27So, you know, there are downsides to this as well.

20:30But no, I always wanted to invest in test coverage, and documentation.

20:35And by the way, and this is an interesting side story we may not have time for,

20:39but like, as a result, you build lots of tools to help you do that too, right?

20:44So I've now got tools that help me generate the documentation, tools

20:48that help me generate test cases.

20:50The documentation, as you identified, has lots of examples and sample code.

20:54Each piece of sample code is also a unit test, right?

20:57It'll actually take the code out of the documentation and run

20:59it which is obviously stuff that I had to go and build myself.

21:02but that was all me, you know, thinking, well, I don't want to, I

21:06don't want a piece of code in the documentation that doesn't actually run.

21:09Like, you know, what happens if I refactor the code and I

21:11forget that it was in the docs?

21:13I know exactly what you're talking about.

21:15There's a little thing that comes to mind.

21:17I'm not sure where I've seen it, but I think it was along the lines of

21:21like a yuck yuck tree, like the, the concept of like yuck shaving and

21:27like a tree of dependencies or like how you would try to do one thing.

21:31And to do that, you had to like diverge into a different side

21:36project and then your side project has a side project and you see like

21:40a dependency tree of Yuck Shaving.

21:43So I know exactly what you're talking about.

21:45That's right.

21:46That's right.

21:47And each one of those tools for the tools for the tools, each one

21:50of them has to be perfect as well.

21:51Right?

21:51So, anyway, it's probably a fractal thing.

21:54, but yeah, that, that, that itself has been a really interesting journey trying

21:57to build the, the software that I really always yearned to be able to,

22:03to build even as it scales, which is, you know, tricky to be honest.

22:07And obviously once people start using it too, because then now you've got

22:11opinions coming from people outside saying, Oh, well, you know, you

22:15might think it's perfect that you do this, but we don't like it like that.

22:18You're like, ah, now I've got this dilemma.

22:20Like, am I going to break my perfect thing to do the thing

22:24that actually a customer wants?

22:25Well, of course, you know, because if no one's using it,

22:28then then what was the point?

22:29So there is a a whole set of interesting trade offs that

22:33you then start, start making.

22:34But no, just again, true to its name, I would say the most interesting

22:38part of that is how did, how do you make it as small as possible?

22:43and I, maybe that's not so important these days in the world of, faster internet

22:46connections, et cetera, and good bundlers.

22:48But like, I wanted to see, like, could I make this like as almost, you know,

22:53infinitely as small as I could make it?

22:55and so I spent a lot of time thinking about minification,

22:57a lot of time thinking about.

22:59you know, how to refactor the code in a way that, you know, made it compress

23:03really small and, you know, it's still, it's still pretty small for what it does.

23:05so I'm pretty pleased with it.

23:06That is awesome.

23:07I highly encourage everyone who's watching this to also check out

23:11your beautiful website and the documentation, the examples, et cetera.

23:16I think you've laid it out very clearly.

23:18but given this is a podcast where most people listen to, would you

23:23mind in words, trying to describe what TinyBase is, like, for example,

23:29in the context of a React app.

23:31Okay.

23:32Well, firstly, TinyBase.

23:34org will do a better job of explaining it than I will do now,

23:38so definitely go, go check that out.

23:40, it's very nice that you say it's beautiful.

23:41I'm no graphic designer and I actually think the website

23:45could do with a bit of love.

23:46So if anybody would like to come along and make it look a little nicer,

23:49they are very welcome to file some.

23:51CSS pull requests on me, I would be very happy to take those.

23:54But, that all said yeah, look, the basic idea or the strap line

23:59is that TinyBase is a reactive data store for local-first apps.

24:05And I'm very careful to say data store, not database.

24:08I don't want anyone thinking that it literally is a database.

24:11it's a data store, but it just has characteristics that are perhaps

24:15familiar to people who have thought about relational databases in the past.

24:19I also say that it's for local-first apps because I'm riding on the back

24:24of the cool local-first movement that's happening right now.

24:27I guess there's no obligation to only use it on local-first apps.

24:31I think, you know, even a permanently online app that wants some tabular

24:35data store in the browser would still be able to use TinyBase.

24:41But yeah, that's the principle.

24:43It's in memory.

24:44So the data is either loaded up from some local you know, to the browser,

24:50some local storage, or session storage, or IndexedDB, and then

24:55it's loaded into memory, and then you build your app on top of it in

24:59a way that is fully reactive to the changes that happen to that data store.

25:04So, in a way, you have delegated all of the data provisioning, loading,

25:11synchronization, whatever, to TinyBase, and you just build your app as a pure

25:16function of the TinyBase contents, and I will take care of getting all that

25:20data updated in the background for you.

25:22Whether I'm pulling it from the web, or I'm synchronizing it with CRDTs, or I'm

25:27pulling it out of a local store, or I'm firing it over PartyKit WebSockets to your

25:32friends you know, I'll do all of that.

25:35All you have to do is build a UI that hooks into tables or rows or cells or

25:41key value pairs in, in the TinyBase data.

25:43and your app will always be reacting to, to those changes.

25:48So that's the basic principle.

25:49It doesn't have to use React, incidentally.

25:51Like, there's nothing to stop you just, you know, listening to changes

25:54in the data directly, and then just, I don't know, document dot

25:57writing out stuff if you want to.

25:59But React does work well because, obviously, that also has a

26:02sort of reactive principle.

26:03And so you can use hooks to attach your components to any of the

26:09elements of the, the, the store.

26:11And by the way, you can do that at any granularity.

26:13So you could have a component that was just watching one cell of one row of one

26:17table, or you could have a component that was watching a whole set of tables, or

26:21one set of key value pairs, what have you.

26:24Lots of different ways to listen.

26:26And you know, TinyBase will make the, or do the job of, you know, alerting you

26:31when just that thing that you've looked, you're looking at has, has changed.

26:35And the same then goes for queries against the database, by the way.

26:38So if you want to do something more interesting than just watching raw data,

26:42you also want to look at aggregates or you know, relationships between

26:46data, or in fact, just queries.

26:48construct arbitrary queries.

26:50You can then attach your app to listen to the results of those.

26:53And again, I'll take care of all the reactivity behind the scenes

26:57and your app will update when, when when those results do.

27:01So I guess when I explain it like that, it sounds like TinyBase is doing quite a lot.

27:05and that's probably why, you know, we're two years in and it's taken me

27:09quite a long time to get to this point.

27:10But yeah, that's the basic principle.

27:12And I would say, I think this is kind of an interesting approach

27:17to apps, whereby You know, how the data gets into the memory of the app

27:22is kind of, like, delegated away.

27:24And you, as an app builder, now don't need to worry about things like

27:29going on or offline, or things like synchronizing, or things like storing.

27:33Like, you should expect your data storage layer to kind of handle

27:37that for you, and you are then just painting your app out of the palette

27:42of data that is provided to you.

27:46because As I'm sure you know, and I'm sure many of the listeners know,

27:50the acts of synchronization and storage and dealing with all these

27:54network conditions is just horrible.

27:56And rather a small number of people deal with those problems than have

28:00everybody trying to build a local-first app having to re solve those problems

28:03is probably the long way to go.

28:06and so I, I, I guess that's what I'm hoping happens.

28:09And it's not just TinyBase, of course, there are lots of other people in this

28:11space, yourself included, but, you know, many other vendors at this point

28:14who are trying to crack these problems.

28:16and I think it's because it's hard and we know that we have to get

28:19this layer of infrastructure going before the, oh the local-first

28:26ecosystem, you know, really takes off.

28:28I suspect that at the moment we may have more vendors building

28:32solutions than we actually have apps.

28:34That's.

28:35Bit of an offhand assessment there, but, you know, there are a lot of

28:38people trying to solve these problems.

28:40And I think it's because it's genuinely hard.

28:42And what I really hope is that we can make it easy enough

28:45collectively that that people can build these apps much more simply.

28:49And you get the benefits of local-first without having to have a degree in

28:52computer science for synchronization.

28:54That's what we're all trying to get.

28:56Totally.

28:56And, and I think we're getting really close to, to that where I think right

29:01now the de facto, solution for state management on the client might rather look

29:07something like MobX or some people might still like use Redux or Redux toolkit,

29:12et cetera, or, or like React query.

29:15and I think it's hard enough to build an app that works in the right way.

29:20And most developers don't, dare yet to think about persistence

29:25and working offline, trying to make it work, just like that.

29:29And I think a technology like TinyBase can really, raise the, the floor quite a

29:35lot in terms of what a state management solution can give you out of the box.

29:39It gives you all of the things something like MobX does, but also on top of it

29:44gives you persistence, gives you offline functionality, gives you syncing.

29:49So I think we just, yeah, raised the floor.

29:53So in terms of the different responsibilities, I'm curious to hear

29:57a little bit more, and I think what stands out about TinyBase here is is

30:01that you really went above and beyond to make things flexible and pluggable.

30:05So for the persistence path, I think you do allow a whole different range of

30:11different options to persist, whether it's IndexedDB or, or other options

30:15as well as for, for the syncing.

30:17So would you mind sharing a little bit more about the details and

30:21the thoughts you have around that?

30:22Yeah, certainly.

30:23So in the very first instance, I was just jSON serializing the content

30:29of the store and writing it to local storage, or to session storage, or to

30:34file, because you can obviously run TinyBase on a node server as well.

30:38And, that's great, and in 2022 that seemed like the Basically, you know,

30:45when you're thinking about small stores anyway that seems like a fine array of

30:49ways to do it, but Obviously people start to push the boundaries of what you can

30:53store in those media and just serialize JSON isn't necessarily the way to go

30:56So I think IndexedDB was probably the next one that I tackled was not as easy.

31:00, just Just as a small side note, IndexedDB is not the most

31:05enjoyable API to work with.

31:08Most notably, it lacks any form of reactivity whatsoever that I know of.

31:12I think there were some proposals that never went anywhere.

31:15But there's no observability.

31:17So it's kind of Crappy to have to poll for changes, but , yeah,

31:22I do now have that as well.

31:24So you can either save your in memory store to IndexedDB or load

31:29it or automatically save to it when changes happen or automatically load

31:33from it with a polling mechanism.

31:35so that, that, that, that's there as well and and then around the same

31:40time, in fact, it was Matt Wonlow that I know it was on your show a

31:43little while back he was working on his CR SQLite solution and I started to see

31:50that SQLite was starting to emerge.

31:53as as a browser opportunity.

31:57And so I figured, well, you know, there'll be people who want to store it in the

32:00browser on, on a SQLite database as well.

32:03So I went down the path of supporting that.

32:06So the tables that were in your memory store in TinyBase can

32:10now be literally written out to tables in a SQLite instance.

32:14Or read in, and again, reactivity is not perfect.

32:19, some APIs are better than others, but you know, do what I can to, to

32:23keep that as synchronized as possible.

32:26And that then opened the door to support things like PowerSync

32:31and, uh Turso and ElectricSQL.

32:37, I also did.

32:38PartyKit along the way, which is not a SQLite store, but it's a

32:42kind of a WebSocket based way of storing data up in a durable object,

32:46and so it supported that also.

32:49So yeah, there was a period, probably over the last year or so, that the

32:52version 4 series of TinyBase, if you like, where each major release was me adding

32:56support for one of these new storage technologies or synchronization platforms.

33:02The other two that are very worthy of note are AutoMerge and YJS, so those

33:08are more classic CRDT technologies, but you can store TinyBase into a

33:13Yjs document, or load it from a Yjs document, or into an automerge document.

33:18And then, of course, you've got the benefit of those two

33:20platforms synchronizing in whatever way they want to.

33:23So, yeah, I guess you could say I, am providing a lot of options.

33:28Maybe too many.

33:29I, Suspect that when you go to the website and you see all these

33:31baffling options, you're like, Oh yeah, but which do I actually choose?

33:34so what I'm hoping to do actually fairly soon is write a guide for, you know,

33:38this is the kind of app I'm building.

33:39This is the kind of platform I should use, or this is how I should synchronize.

33:43Do I do the synchronization myself or do I rely on some third party to do it?

33:47And at the same time, obviously we've seen companies like PowerSync and

33:51ElectricSQL, you know coming forward with their, their solutions here.

33:54And so, yeah, if people are using TinyBase and they want to Provide

33:58a a gateway to those systems, then that's that option as well.

34:01So yeah, I have tried to be as flexible as possible.

34:04One of the privileges, I guess, of working on this as a hobby is that,

34:07you know, I can be open to working with as many of these partners as possible,

34:12whether they're commercial startups or just open source projects, right?

34:15I'm not in competition with anybody.

34:18And so if I'm providing a way for developers to onboard onto these

34:21different things, then that's great.

34:22I definitely have a view that the, the tide is rising up.

34:26I'm saying that on a boat but the local tide is rising, local-first tide is

34:30rising, and I'm happy to see all boats go with it even in the long term.

34:35, even if in the long term TinyBase becomes not needed because, you

34:40know, all of these other platforms have, have matured to the point where

34:43they're offering what I already do.

34:44But I think in the meantime, I'm hopefully providing a nice, disambiguation

34:49layer or you know, something like that to help people decide what their

34:53approach is going to be without having to rewrite their app dramatically.

34:57So yeah, today you could store your TinyBase data to local

35:00storage and tomorrow you could then sync it to ElectricSQL.

35:03Why not, right?

35:04So in terms of the data that is persisted, let's say to IndexedDB or OPFS, SQLite,

35:11or synced to one of the syncing providers, how do you bring the data into memory?

35:18And do you typically, let's say I have a, let's say I'm using

35:22Turso and have my SQLite database somewhere, Is TinyBase automatically,

35:27hydrating all of that data from the database into the in memory version?

35:32Or is there some sort of scoped version where maybe the database is two

35:36gigabytes and you only can constrain yourself to 200 megabytes in memory?

35:42how are you handling that?

35:43Right.

35:44So for all of the database based persisters, I call them, there's a

35:48configuration where you say which tables you want and, you know,

35:52whether you want to just load from those tables or whether you also

35:55want to be able to save back to them.

35:57the worst thing I could think of would be somebody with some large production

36:00database and they connect TinyBase to it and the next thing you know, it's

36:03tried to load the entire database into memory or it's tried to write No, back

36:08to the whole database, and then, you know, that's the end of the world.

36:10So no, it's all very configurable on a table by table basis.

36:15And the one thing I have not done, but I'm pretty sure I need to

36:18do, is also provide some kind of pagination or filtering on that view.

36:23because if you've got a database that has, let's say you, you, you shard

36:27it on user or something, and you only want to be able to see just the data

36:31from that user, which, you know, it's gonna be a pretty common use case,

36:34then make sure that persistence is not for the whole table, but just

36:38for the relevant rows, of that table.

36:41I think it's going to be harder than I think it is.

36:43I suspect there are lots of gotchas to doing that.

36:46, and so for now, there's my, my solution or my suggestion will be that people

36:51doing this should probably have a per user database, which is kind of an

36:56interesting approach that I know some people have started taking anyway,

36:59for, for edge based, databases.

37:02Um, so that, you know, you're, you're in no danger of, accidentally

37:06loading data from a different user into someone's browser.

37:09But no, that's definitely something I need to tackle.

37:12So it's scoped, to answer your question, it's scoped to individual tables as

37:16to whether it's load or save or both.

37:18But I think over time we need to add more pagination.

37:21So that you are just looking at, you know, the top 10 records,

37:23100 records, or what have you.

37:25The reactivity gets really tricky at that point because I'm sure you

37:29know, you know, SQLite's reactivity out of the box is pretty weak.

37:32At best, you're going to know that a table changed.

37:35And what are you going to do?

37:36Query the whole table to find out which row it was that changed.

37:40And so knowing that it was row 47 out of 2 billion that was the one that changed.

37:45is not currently something that the database platforms provide by default.

37:49Um, So, that's one thing where I am really having to work around

37:54those limitations right now.

37:56, and I'm hoping, if I can't make it clearer to you, I'm hoping that people

37:59working on SQLite are going to crack this and provide, you know technologies

38:03that make it easier to synchronize at a much more granular level, or get

38:06reactivity at a much more granular level, so that TinyBase isn't having to poll

38:10everything in a way that doesn't scale.

38:13Yeah, I mean, I, I think, Ideally for SQLite, the most principled and

38:21foundational approach would be to rebuild SQLite from the ground up to

38:27be incremental and to be reactive.

38:29And I think actually a former coworker of yours, who is um, Julian, who

38:35has also created, like HipHopVM at, at Facebook and, and HackLang.

38:40he's actually trying to, to do that with a new startup called, Skip that is a

38:45very ambitious, or even audacious goal to build the whole new database from scratch

38:50to be reactive, from the ground up.

38:53Which I think is very, very interesting, but short of that, I think you, you're

38:57left by imposing, reactivity, primitives on top of SQLite and the best you can

39:04do there is to do the minimal amount of work, you need at a given situation.

39:09So instead of pulling the entire table that, you know, okay, row 47 has changed.

39:16And that requires quite a bit of like a reactivity system on top of it.

39:20And so luckily with, a signal like system, that is a, it's a pretty

39:25good, primitive to put on top of it, to implement that reactivity.

39:30But, yeah, it's not as efficient as you, if you'd build it

39:33from the, from the ground up.

39:35That being said, SQLite is so fast that you get a lot of performance benefits out

39:40of the box and you can get away with it.

39:42Yeah, that's true.

39:43you know, certainly on the client.

39:44Um, yeah, I, I, without, you know, sharing too much, internal stuff,

39:49there was a big push, at Facebook.

39:51I'm thinking six, seven years ago, you know, to think about reactivity for

39:56the entire Facebook stack, because this concept of, you know, taking the ideas

40:00of React that were out on facebook.

40:02com and bringing them all the way back through the web servers, which are

40:06HHVM, all the way through to the data stores that were being used at the

40:10company, you know, like it's a utopian vision, but like, if you can make

40:14that push based all the way through.

40:16That's spectacular.

40:17and you know, Julian and I were probably in many of the same, uh

40:22discussion groups and, you know, internal Facebook groups where a lot of

40:25these ideas were being bounced around.

40:27And, and so yeah, when he left and went off and did, Skip and you know,

40:31I guess maybe some of this informed my thinking about TinyBase too, you know,

40:36that dream is still very much alive.

40:40and yeah, it's coming from, ironically, it's like it's coming from React.

40:44That idea is coming from React.

40:45It's just like, well, okay, can we bring that idea a little bit further back,

40:48and then a little bit further back, and then a little bit further back?

40:51Oh, like, and then we end up on the database.

40:53Like, why can't we get the database to be, you know, telling us about these changes?

40:57And, yeah, I guess that would be the vision.

41:00Unfortunately, I haven't checked in recently, but I think SkipDB wasn't

41:04Wasn't at a level where I could start working with it, when I last looked.

41:08Um, so yeah.

41:10I think it's not yet fully production ready, but as far as I know,

41:13they're just still working on it.

41:15So in terms of the query layer for TinyBase, you have, not

41:21implemented SQL as a query language on top of your own data store.

41:25Um, so yeah.

41:25But you've provided, given that JavaScript or TypeScript is the

41:29primary way how you interact with it, you've just embedded a DSL, as a

41:34query language directly into TinyBase.

41:36So would you mind briefly describing that and, how that describes a user

41:40experience or the developer experience?

41:42Right, right.

41:43So the, the journey here for me was that first I knew there were going

41:47to be some very simple query like primitives that I wanted to have.

41:51So one is, like a metric, which is some kind of, you know, number

41:57derived from the content in a table.

41:59And I knew that that was going to be the first thing I wanted to have if I was

42:02building this hypothetical GitHub app.

42:04I was going to want to have a number in the top that said number of repos,

42:07and I was going to want to have a number that said number of commits.

42:10And so I actually baked in.

42:12a kind of a metrics oriented DSL where you can define a metric which is, you know,

42:19basic aggregates, counts, averages, sums, those sorts of things, which by default

42:24is normally just counting the rows in a table, but that is then a reactive entity.

42:29So again, you can say, I just want this span in the top right hand

42:33corner of the window to just always show whatever that metric was,

42:36and then you can forget about it.

42:38And if that table is updated, You know that number's going to change,

42:40TinyBase will take care of it.

42:42another thing that I knew I was going to need to do was

42:44relationships between tables.

42:45So I built a very specific thing for just one local table, one remote foreign

42:51table, and then like keys between them.

42:53So the value of one column in one table is used as the identifier for another.

42:57And so, yeah, you want to see the issues just for this repo, or you want to see

43:01the repos just for this user, then you'll be able to do those kinds of things.

43:04So those are, were actually baked in pretty early to TinyBase, just because

43:07they seemed Very important, but I, I guess I knew eventually people were

43:13going to want to push that on and start doing things like arbitrary queries.

43:20And I toyed with the idea of, you know, supporting SQL in some form.

43:26And I should say, I love SQL.

43:28I have been writing SQL most of my professional career, which

43:30you know, goes back a fair way.

43:33And my most recent role at Facebook was, you know, the data analytics

43:37team in the data infrastructure org.

43:40So you know, we were working with SQL all the time.

43:43Perhaps despite that, or because of that, I kind of got nervous about doing

43:47an arbitrary SQL parser, evaluator, executor, and then being able to make

43:53the results reactive, which was like, that's the, that's That's the non deniable

43:58requirement, like, it has to be, reactive.

44:01So, I didn't really have a lot of choice, but, oh, and by the way, you

44:06know, a full SQL parser with all the dialects you'd want is, like, it's

44:09gonna quadruple the size of TinyBase.

44:12It's just, it's code based.

44:13So, no longer tiny.

44:15Yes.

44:15So, I wanted to come at it from a slightly different direction.

44:19I know I'm not the only person to have thought of doing more like a DSL approach

44:22to this, but I, I wanted to see whether I could capture as much of the, the

44:26valuable parts of SQL without turning it into a place where people could build

44:31stupid cartesian joins and kill their app.

44:34so I wanted to make it kind of a little bit on rails.

44:37And yeah, it was the idea of TinyQL was born, which is kind of a stupid name.

44:41But it's, as you said, a DSL that allows you to, you know,

44:45select, where, group, limit.

44:48, actually, no, I don't have a limit in there.

44:50I limit a later part in the pipeline.

44:53But, yeah, basically allows you to join tables together.

44:56and then where them, there's havings and groups and those kinds of things too,

45:02which like really is what 98 percent of people want to do with SQL anyway.

45:07And haven't really come across too many queries that I can't express

45:11with these five or six quote keywords.

45:14Yeah, the cool thing about SQL is you can do anything.

45:17The awful thing about SQL is you can do anything.

45:19And, I have worked with enough people who thought they knew what SQL did and

45:24didn't quite know what it did to, you know, see the trouble you can get into.

45:27So I really, yeah, wanted to make it a little bit more constrained than that.

45:31And it's, it's been, it was a real journey.

45:33That was a very tricky, for me, anyway.

45:35, that was a very tricky thing to build, especially making sure

45:38that all the results are reactive.

45:39Because, by the way, you know, any of those tables could change.

45:43Any of the rows in those tables could change.

45:44In fact, you don't want to react to the changes of the results if

45:47some of the irrelevant rows change.

45:49You only want to know if it was the relevant rows that changed.

45:51So you have to set up listeners on all the things that were relevant and

45:54not the things that were irrelevant.

45:56so that ended up being a little harder than I thought, but yeah,

45:59pretty proud with how it turned out.

46:00And as a result, and I would urge people to go check this out, You can

46:04build kind of interesting analytics apps, right, with this, in the browser.

46:09So if you go to the TinyBase.

46:11org demos, you'll see there's a demo.

46:14I think it's called car analytics or something like that.

46:17And it takes a data set of car data, different models, different

46:20makes, miles per gallon, years.

46:23So there's a bunch of numerical measures and there's a bunch of dimensions.

46:27And in the browser, it loads all the data in pretty quick.

46:30It's not that much data, but you know, loads it in pretty quickly.

46:32And then you can do groups, sorts, filters, across all

46:35of this data in the browser.

46:37And it's like, it's completely instant.

46:39It's like, great, love it.

46:41and look, this is not going to be great for querying the Facebook data

46:44warehouse, however many hundreds of petabytes that is these days, but

46:49it is going to be fine if you've got 10 or 100, 000 rows in your browser.

46:53And Don't be scared, people.

46:55Like, getting a hundred thousand rows of data into a browser is not hard.

46:58The browser is perfectly capable of doing that.

47:00You know, most even phones are very happy to run that kind

47:04of sized data in, in memory.

47:06so yeah, for a lot of even analytics use cases, you can do, you could, you

47:09could build interactive dashboards with this kind of technology and

47:13have a sort of constrained set of queries all running off TinyBase.

47:17And it's super feasible, and it's been.

47:20It's been fun to build that.

47:22Whether or not people are using it in the wild, I don't know.

47:24I should say, I don't put any instrumentation into this product.

47:27So who knows what people are using it for, or how they're using it.

47:31But I hope that those people who use it do, do find the value of being

47:34able to run queries, without, shooting themselves in the foot with SQL.

47:37Did you follow any sort of prior art in regards to how you built

47:41that reactive query system?

47:43So, as I was working on a similar part as, the work on Rffle, there,

47:48there was, some prior art in regards to Adapton and mini Adapton.

47:53And so this was, some inspiration, that we followed for the reactivity

47:57system and we've like over time learned that, it's very similar to how now like

48:04signals is all the rage these days.

48:06It's like very similar in terms of the ideas.

48:09So did you follow some, some similar paths?

48:12No.

48:13And I wish I had.

48:14I think it's one of my weaknesses.

48:16One of my weaknesses is that I look at a problem.

48:18You know, it's like the famous Hacker News thing.

48:19It's like, I could have built that in a weekend.

48:22I, I look at a reactive query system.

48:23I think, yeah, I'm sure I could build that.

48:25And then like two months later, I was like, I wish I'd read a bit more.

48:30I am not good at prior art and reading scientific papers or even

48:35just, you know, looking at other projects to see how they're done.

48:37I think, you know, I know what the result should be and I'm

48:39going to try to build it myself.

48:40And then I get into trouble.

48:42Fortunately, I worked my way out of trouble.

48:44If it makes you feel any better, even with knowledge of the prior art,

48:48it still took easily more than two months just to get anything working.

48:53So,

48:55Okay, that's just, that's just my, my weakness that fortunately I

48:59was able to work around this time.

49:00But yeah, I think the other thing is that I obviously knew in intense detail

49:06how the reactivity of the underlying tables, rows, and cells were right.

49:11And so I knew how I was gonna have to cascade the listener trees into all

49:16the relevant parts of the database.

49:18And, That was, yeah, that was just something I had to figure out for myself.

49:23The other thing I alluded to earlier is that TinyBase is

49:27just draped with test cases.

49:29Like, I am exhaustive, overly exhaustive.

49:32I'm completely OCD when it comes to test cases.

49:35It's not quite TDD.

49:37I don't always do the tests first, but, you know, I do eventually try

49:41to make it look like I did TDD.

49:42So I built every possible query I can think of, and I build all these demo

49:46apps, and I build all these sample, you know, fragments of code, and

49:50they all get tested exhaustively, and so I figure out, well, once, once,

49:55once those work, I'm probably there.

49:57So that, that was my benchmark, and I got there.

49:59It's probably, I'm sure it's still possible to break it,

50:02but, you know, I think, I think I'm about where it needs to be.

50:06So one of my favorite problems to, to think about, and it's a

50:10love hate relationship in terms of technical problems, which are schema

50:14migrations and schema evolution, which is something I thought a lot

50:18about, through the work on Prisma.

50:20And I tried for the work on Overtone, ideally having to deal with as little

50:27as possible, or at least have a very principled foundation for that.

50:30And I feel this is a bit of a underdeveloped area in some

50:34of the available open source technologies right now, or

50:36local-first technologies right now.

50:38So I'm curious how you have been thinking about schema migrations for TinyBase.

50:45and if you've already built some foundation there.

50:47Okay.

50:48Well, before we talk about schema migrations, we should talk about schemas.

50:51So.

50:52even having a schema was kind of a late thought for me, TinyBase outta the

50:57box by default, when, and especially when I first built, it, didn't have

51:00the concept of a schema at all.

51:03And, you know, any row could have any set of cells.

51:07I mean, a, a a row of cells was basically just a map.

51:10So it was possible for every row of a table to have completely different

51:14cells in it, which is a bit.

51:16Scary, and contrarian especially for someone like me who'd

51:21just come from working on the world's largest data warehouse.

51:24the idea of doing this was, was a little daunting.

51:26But,, I felt like, you know, the word tiny meant I was allowed to do things like that

51:32and take away some constraints and just let people put arbitrary data into this

51:38basically a map of map of maps, right?

51:39Tables of rows of cells.

51:41Now that obviously is fun for little demo apps, but at some point you

51:46need to start tidying things up.

51:48It's all very good when it's just one person building it too, right?

51:50Because I can remember all the fields that are supposed to be in this table.

51:53but when I've got colleagues who maybe want to know what the schema is,

51:55then should do something different.

51:57So I did, yeah, we do have schema support.

51:59I did add schema support, but it's optional, right?

52:02You, you, you have to explicitly add the schema onto your store.

52:06but even then the schema is, I'm afraid, sorry, very basic.

52:10So you can say whether a cell is a, you know, string, a boolean or a number.

52:13And you can say whether it was required or not, and what the default is.

52:17That's it.

52:18So that seems a little daunting.

52:20I should say, by the way, that TinyBase only supports those data types anyway.

52:24So you can't put anything other than a string, a number, and

52:27a Boolean into a field anyway.

52:29So I don't have support for arrays, or objects, or even null.

52:35so that kind of makes you realize this is a relatively simple thing to model anyway.

52:41But yeah, that, that, that's, that's where we are right now with schemas.

52:46And once you've added a schema, it will try to massage the data

52:51into that schema, and it'll drop anything that doesn't match.

52:54that's pretty basic, I will admit.

52:55But there are two, I think, two main things I need to do from here with this,

53:00because clearly that's not sufficient.

53:01It's great for fun, simple apps.

53:03It's actually, I will say, it does allow you to do more than you might think.

53:07But anyway, people do want to do arrays, and That lets you do many to many joins,

53:12and they do want to do objects, so fine.

53:14We should probably get to that at some point.

53:16So, that's, yeah, that's my big push, I think, is making time based support

53:20richer data types inside a cell.

53:22I think arrays will be the first.

53:24I'll probably stick to having arrays of one, type, like it's going to

53:27be all numbers or all strings.

53:29I don't think I'm going to have mixed arrays.

53:31That could get too tricky.

53:32I still want to have some kind of validation.

53:34so we'll do that.

53:35, that knocks off a many to many requirement that a bunch of people have had on GitHub

53:39that I can't currently do very easily.

53:41And so we'll, we'll do that.

53:43Secondly, I would love to do more with the schemas themselves.

53:48So the, schema dialect that I put in right now is just, like I said,

53:53a fairly simple description of what the rows are, but There's Zod in the

53:57world, and there's Effect, and a bunch of other ways that people like to

54:01express schemas, and if they've already built their schema for some other part

54:05of the app, or they've built it for some, you know, some other part of the

54:08system, maybe up on the server, like, how can they then turn that into the

54:13schema that resides in the app itself?

54:16So I have no doubt that I will be building Zod to TinyBase Converter, and Effect

54:22to TinyBase Converter or some, some, something to map, map those schemas in.

54:27Still doesn't solve your problem of migration, and I will admit

54:30that I am a bit daunted by that.

54:34In general, I am daunted by the whole space of anything that

54:36requires build time code mods and, you know, running generated code.

54:42Like, I just hate that because immediately you've got a team of 10 people and all

54:46of their artifacts get out of sync.

54:47I've seen that at Facebook, right?

54:49Whether it's GraphQL or what have you.

54:51So I, always gonna be trying to look for solutions that don't require you

54:55to run a build step, to generate some magic code that you then have to remember

55:00to link in, how to actually avoid that step and still do schema migration.

55:05is not quite in my head yet.

55:07So I share your sentiment.

55:09These things are underdeveloped, but I don't have an answer for you.

55:12Right.

55:12I can share my thoughts on it, which is, kind of like an architectural one, that

55:18has quite a lot of implications, but those implications are appealing to me, I would

55:23separate the read path from the write path and have the write path be event driven.

55:29And that allows me to do sort of event sourcing.

55:33So similar to how Redux works, but, persisted and distributed

55:37Redux, if you, if you want.

55:39And, then like the way, how you interpret your Redux events.

55:44you can interpret them however you want, and you can very

55:47easily change that over time.

55:49So you basically just go through the entire event source, the entire event

55:53log of events, the entire history, and your app database that you

55:58read from, that is just, a result.

56:01By going through all the events and every event, you might come

56:06across like a user signed up event.

56:08, you then translate into a insert into users with all the values.

56:15And you can change the user table at any moment, you can, instead of

56:19having one user table, you can have a customer's table and an employee's

56:23table and just reinterpret the events.

56:26And so this way, you don't have schema migrations at all.

56:29You just change sort of the . Interpretation of your event.

56:33it requires a bit of a different architectural approach and I'm

56:36building LiveStore all around this idea, which is very different

56:40from, from other approaches.

56:42But it's the most principled approach that I can currently think of in

56:47terms of schema migrations, where you don't do like my schema migrations,

56:51basically just I design a new schema.

56:53And I target that and there's no, like, up or down scripts.

56:58and so that is very appealing, but it has other downsides as well.

57:02But this is certainly an idea that I, that I think is worth exploring more.

57:07No, I like that.

57:08And it also shares, , Similarities with what I'm separately having to do

57:12for the, native CRDT synchronization, as of TinyBase version 5, which at

57:18the time we're speaking isn't quite released, but by the time this podcast

57:21is out might have been released.

57:22So, hey everybody, go download version 5.

57:25but no, it's, it's got a native CRDT support and I do that by sending,

57:29you know, basically little, little messages, but those are the data.

57:33level, right?

57:33It's, you know, here's, here's how this cell changed, and I wonder whether

57:37like there's some higher abstraction for those messages that is then sort

57:41of schema agnostic or that can be mapped into whatever the schema is.

57:45Because by the way, the minute you start synchronizing things, the schemas

57:49getting out of date is going to become an issue very quickly, because maybe

57:53you've updated the app on your phone, but you haven't updated the app on

57:55your desktop, but now they're jogging.

57:58You know, WebSockets to each other and all hell breaks loose because

58:01the schemas weren't quite the same.

58:02It's all questions at the moment.

58:04Yeah.

58:04I think there's a whole episode just to be done around that.

58:07Happy to form a little working group around this, but I think that goes

58:11beyond the scope of, of this podcast.

58:14but yeah, I think you've come really, really far with TinyBase and this

58:18is, I think, a very common area of the local-first technologies part

58:24that is not yet as fully explored.

58:28and I've just changed my priorities to go double down on that from

58:32the beginning, since I know like how gnarly this is to deal with.

58:37Well, can I just say that, you know, from the point of view of exhaustiveness,

58:41there are many, many unsolved problems in the local-first space.

58:46And I have a horrible feeling that it's easy to think that

58:50synchronization is the problem.

58:52And then once you've solved synchronization, you're good.

58:55Well, I don't think that's true.

58:57And the reason I think that is that I've tried building some local-first

59:03apps that, you know, have things like anonymous versus authenticated access,

59:08or that have private data versus shared data, or that have online, offline, and,

59:14you basically multiply this state machine together and you've got like, you know,

59:18two to the power of four combinations of state that your clients can be in.

59:23And.

59:23It's all very well being able to synchronize data back and forwards, but,

59:26like, you don't want to synchronize a non shared store with another user, or

59:29you don't want to share an authenticated, you know, you brought a copy of an

59:34authenticated database to be local, but now you log out and now that's

59:38got to go, like, the state machine of what it actually takes to build a

59:40real app like Figma or I don't know.

59:45It's like, wow, there are lots of crazy, crazy transitions that can change here.

59:49And I don't think anyone's really thinking about all of those.

59:52I think we all got hung up on, oh, once we crack CRDTs, we'll be good.

59:56No, no, no, no, that's the easy part.

59:57It's not state management, it's the state of the state management in a way.

1:00:00It's like, should I be synchronizing this data?

1:00:03Because this person isn't logged in, or is logged in, or is online, or is

1:00:07offline, or has shared this document, or hasn't shared this document, and

1:00:11that is actually unsolved, I think.

1:00:14So I built a little sample app called Tiny Rooms a way back, and like,

1:00:19the app itself is stupid, it's just dragging rectangles around, but like,

1:00:21I tried to model all of these different states, and it was really hard.

1:00:25Like, I was, oh god.

1:00:27I would hate to be actually trying to build a production grade

1:00:30local-first app at the moment because those problems are really hard.

1:00:33Yeah, I think the, the benefits are that if you're building linear

1:00:39or if you're building Figma or if you're building, Tiny rooms.

1:00:43the advantage for you is that you have very specific trade offs, very specific

1:00:49constraints that work for your app.

1:00:52And so you can take tons of shortcuts that for a more

1:00:55general solution are not viable.

1:00:57But for your app works.

1:01:00And so, given that, well, there, there's many trade offs that

1:01:04Figma can make that would not work for Linear and, and vice versa.

1:01:08But I think that is where you can still build things in a local-first way,

1:01:12that is like competitively, beneficial.

1:01:15but I agree having the best of all worlds, is where we're a few

1:01:20days away from that, at least.

1:01:22Yeah.

1:01:23So I think that's one of the curses of being a, a tools

1:01:25builder like, like you, right.

1:01:27you know, I feel that I'm building shovels for people who are going

1:01:30to go off digging for gold.

1:01:31but I have to build a shovel that'll work in all different

1:01:34types of terrain to be successful.

1:01:36Um, and so like I said, you know, is it going to be online, offline,

1:01:39auth, not auth, stored locally, stored remotely shared or not shared?

1:01:43Like, okay, now 16 combinations, got to support all 16.

1:01:46Whereas yes, if you're Figma, you can say, well, no, you

1:01:48can't ever use it anonymously.

1:01:49You've always got to be logged in.

1:01:50It's like, oh, great.

1:01:50Okay, fine.

1:01:51Like, I've just reduced that problem to eight.

1:01:53Yeah.

1:01:54Pick your trade offs very carefully.

1:01:56And so this is why I'm also, why I'm working on Overtone

1:01:59and LiveStore at the same time.

1:02:02So if your app is sort of like Overtone esque shaped, then LiveStore

1:02:08will work really well for you.

1:02:09If it looks rather like Facebook, like a social network, or has like

1:02:13some very different shapes and trade offs, then it's not as good of a fit.

1:02:18But I think this is also what's so nice about having all those different

1:02:22local-first tool vendors is they have all their different starting points and all

1:02:26their different thoughts on trade offs.

1:02:29And, I think this is where we're going to have a great shovel

1:02:32for many different terrains.

1:02:34Yeah.

1:02:35I think that's exactly right.

1:02:36You know, it's a dilemma.

1:02:37Do you build a tool that fits one niche perfectly?

1:02:41And, yeah, of course, that really informs, and you'll build a, you know, brilliant

1:02:45tool for the one job you have in mind.

1:02:48but at the same time, you may have limited your market.

1:02:51Now, fortunately, I don't have to think about those things because, you know, it's

1:02:54not a commercial, commercial business.

1:02:56and I'm, but I'm trying to keep as many People are happy as possible.

1:02:58And if I'm just listening to the feedback that's coming in on, on

1:03:01GitHub and people are saying, Oh, I want it to go off in this direction.

1:03:04And, Oh, someone else wants to go off in that direction.

1:03:05It's like, okay I'm going to try to, maybe I'm going to

1:03:10try to keep everybody happy.

1:03:11but then the complexity falls onto me.

1:03:13And that's where being a tools builder starts getting hard work.

1:03:16You know, this.

1:03:18Right.

1:03:18I mean, for, for what it's worth, I think I'm trying to actually, write

1:03:23down the cases where LiveStore is not a great fit for, and I'm trying

1:03:27to be like, very explicit about the trade offs that I'm making.

1:03:31So for example, LiveStore right now expects you to load all of your data

1:03:35into a single SQLite in memory database.

1:03:39Luckily, SQLite is very efficient, like based on my benchmarks and testing, much

1:03:44more efficient than like JSON in memory.

1:03:47It's much more efficient to keep like hundreds of thousands of rows in,

1:03:51in memory SQLite as you keep them in just JavaScript objects in, in memory.

1:03:57you need to just, deserialize the, just a few ones that you

1:04:01actually want to like currently keep in memory for, for your query.

1:04:05And so, but that's a major trade off.

1:04:07If you have your database, if that's necessarily five gigabytes.

1:04:11Then LiveStore is probably not a great fit for you.

1:04:14Or if you don't want to follow the more like event sourced, nature of it, then

1:04:20it's probably also not a good fit for it.

1:04:21But I think this is what makes it more specific for when it's a good

1:04:26fit for someone, whereas if it's not.

1:04:28So actually this brings us onto an interesting thing that I think

1:04:31this community needs, which is.

1:04:34kind of a, a community view of what all these pros and cons are of the

1:04:38different solutions and what the decision tree should be if you want

1:04:43to build an app of a certain type.

1:04:45You know, am I going to be online or am I going to be offline?

1:04:47Am I going to be authed or not authed?

1:04:48You know, how do I work my way down to what is going to be the best solution?

1:04:52Because of course, you talk to a vendor, they're going to say,

1:04:54oh, well, we can do everything.

1:04:55but something that allows people to know when, you know, when should I use.

1:05:00In something like the more mature front end world, people know when

1:05:04they should be using React versus Svelte or whatever, I guess.

1:05:07There are plenty of places you can go and read up about what the pros and the

1:05:12cons are of these different solutions.

1:05:14But right now, if you're looking at Database Sync Technology A and

1:05:18Database Sync Technology B, they're both local-first, apparently, right?

1:05:22But one's going to be better for one thing versus another.

1:05:24Who's going providing that commentary.

1:05:26And I think we haven't got that critical mass of users yet to be able to say, well,

1:05:30I tried it and this didn't scale, but I went to this one and this one worked.

1:05:33Someone else saying the opposite.

1:05:34And then you can like have that debate.

1:05:36and so.

1:05:38Actually, this is something I didn't mention, but, you know, one of, one

1:05:40of the parts of my local-first journey was, was putting together a directory

1:05:44of these solutions that I was finding.

1:05:46And so I launched that as the localfirstweb.dev site.

1:05:50And that was partly me just trying to enumerate what these

1:05:53different solutions were.

1:05:55And obviously a community is built up around that.

1:05:57And now we obviously have the conference as well.

1:05:58And what I'm really hoping is that we see more and more people building real

1:06:01apps, learning What trade offs they're having to make, and which of these

1:06:05solutions they're then converging on.

1:06:08given those and, and I think it's fine to have lots of different types of tools.

1:06:11Sometimes you need a pickaxe, sometimes you need a shovel,

1:06:13sometimes you need a pan to get the gold out of the river, right?

1:06:16But it's not always obvious which tool you should take until you've

1:06:19actually gone out into the hills and started looking for gold.

1:06:22Sorry to stretch the analogy, maybe it's because I lived in California

1:06:24for a while, but you know, I think we're not at that point yet, right?

1:06:27We haven't had enough, we've not had enough miners going up into them,

1:06:30there hills to go look for the gold.

1:06:32so we haven't formed opinions about what the right trade offs are and

1:06:35which are the right tool chains to use.

1:06:36We will get there.

1:06:37So speaking of real apps, it seems like I've recently nerd sniped on Twitter.

1:06:42I've recently nerd sniped you into building another app on top of TinyBase,

1:06:48which is a local-first GitHub client.

1:06:51And you've made quite a lot of progress on that.

1:06:53Do you, do you mind sharing more?

1:06:55Sure.

1:06:55So that was a, that was a clinical nerd snipe.

1:06:58That was, that was one of the finest even though I was the victim.

1:07:01So as I mentioned, you know, one of my original ideas for an app that would

1:07:04use something like TinyBase was an open source dashboard that would let you see

1:07:08large numbers of repos and, and, and, and study them and understand them at scale.

1:07:12And I just never got around to building it.

1:07:14And then of course you said something on Twitter like, Oh, wouldn't it be awesome

1:07:17if there was a local-first GitHub client?

1:07:19I'm like, No!

1:07:20Damn it!

1:07:21That's what I was supposed to build!

1:07:22Now I remember!

1:07:23So I took that and I figured that I would maybe go and put

1:07:28my, my code where my mouth was.

1:07:31So I started a little repo called TinyHub, which is in the

1:07:36same , parent org as TinyBase.

1:07:39So I guess you have show notes.

1:07:41There'll be a link in the show notes and, people can go check that out.

1:07:44And yes, it's a GitHub client for running locally in your browser.

1:07:48And I wanted to try and build it as quickly as I could, just to hopefully

1:07:52prove to myself that building apps with TinyBase would be fun.

1:07:56And what it does is it, there's a quick bit of GitHub OAuth right

1:08:01at the beginning, and it then will pull repos that are either starred

1:08:05or that you, have personally, or any orgs that you're a member of if

1:08:09you accept those OAuth permissions.

1:08:12Anyway, long story short, it pulls the GitHub data via the API down

1:08:16into a local TinyBase instance.

1:08:19In fact, a number of instances.

1:08:20It actually has, there's a TinyBase instance per repo, because if you

1:08:24pull React, like, okay, now we've got 20, 000 pull requests or something.

1:08:29So it will pull data locally, stores it in local storage.

1:08:35Yes which seems to be fine, actually.

1:08:38, just, it's a, the first hundred records which is the basic

1:08:42pagination for, for GitHub.

1:08:44So it's, it's not, you know, a comprehensive open source dashboard, but

1:08:48it at least gives you A sample of, you know, 100 repos from 100 orgs and up to

1:08:53100 pull requests and issues from each.

1:08:55And yeah, then just paint a React UI out of that data.

1:08:58And if you go check the repo, you'll see I put a little video

1:09:02on the front page of the readme.

1:09:04And of course, the beauty of this is that now all of this GitHub

1:09:06data is sitting in your browser.

1:09:09And, it's in memory, but you can flip through it at 16 milliseconds

1:09:14a frame, to your heart's content.

1:09:15So, you know, you press the, you press the down cursor and it whizzes down

1:09:19through the pull requests and it's pffft.

1:09:21Views them all instantly.

1:09:22Not a spinner in sight.

1:09:24It's pretty awesome.

1:09:25I'm not sure it's a particularly useful tool at this point because it's read only.

1:09:28Um, and it obviously has to do some work to pull the data down at the beginning.

1:09:33But it certainly, I hope, emphasizes the fact that if you co locate your

1:09:38data with the app, which I think is kind of like what this, this, this

1:09:41local-first thing should be all about at some point, it's amazing.

1:09:44Like, the UX is spectacular.

1:09:46Um, and you can just paginate through this stuff at the speed of thought.

1:09:50Which is really amazing.

1:09:52I will just add a little side note here, is my philosophy around local-first, or

1:09:56my motivation for local-first, apart from being on a boat, is the user experience.

1:10:01I'm just tired of looking at spinners, because I'm waiting for

1:10:03something to pull down and cache locally and then get thrown away.

1:10:06Like, if I can get everything into the browser as soon as possible,

1:10:09and then just go through it at 16 milliseconds That's, that's amazing.

1:10:13So hopefully that has emphasized that point of local-first.

1:10:18Yes, it's about data governance.

1:10:20Yes.

1:10:20It's about, you know, owning your own data.

1:10:22Yes.

1:10:22It's about you know, all the other things that are laid out in many, many

1:10:26other essays other than mine, but like having a UX like that is really, for me,

1:10:31what it's all about, that's the focus.

1:10:33, and yeah, so I'd encourage people to go check out TinyHub and that's

1:10:37me building what I think is a state of the art TinyBase app.

1:10:42I would still say that to make it fully functional, making it read write is

1:10:47a little more tricky because whilst GitHub has some pretty awesome bulk

1:10:51read APIs the write APIs are a little different and making changes to a

1:10:58local datastore and then going back online and reconciling that back up

1:11:01to GitHub is not as easy as it sounds.

1:11:04I'm sure it's Events are probably the answer to that, is what you'll say.

1:11:07Um, but but I'm not there yet.

1:11:09Yeah, again, to my earlier joke that we have more tools than, apps at this

1:11:14point, I really want to just add to the pool of local-first demos, right, but

1:11:19add to the number of apps that show what could be done, and really emphasize what

1:11:23the benefits are of this whole movement rather than just building shovels.

1:11:27I'm actually going to go out and look for some gold myself as well,

1:11:30just to make sure the shovel's okay.

1:11:31I think that's a wonderful place to, to wrap it here.

1:11:35James, thank you so much for coming on the show today, sharing all of your wisdom

1:11:40on TinyBase, the path that led you here.

1:11:43Thank you so much.

1:11:44It's been a huge pleasure and I wish the podcast the greatest success.

1:11:48I'm so excited that you've set it up and honored to be a small part of it.

1:11:52And thank you to everybody that has made it this far and,

1:11:56followed the TinyBase journey.

1:11:57It's been a privilege to work on this and share it with you all.

1:12:00Thank you for listening to the local-first FM podcast.

1:12:03If you've enjoyed this episode and haven't done so already, please subscribe and

1:12:06leave a review wherever you're listening.

1:12:08Please also share this episode with others.

1:12:11Spreading the word about the podcast is a great way to

1:12:14support it and to keep it going.

1:12:16A special thanks again to Rocicorp and Expo for supporting this podcast.

1:12:20See you next time.