Enabling data at massive scale with Next.js

Next.js Conf 2022 / 33:44

Watch this talk on YouTube

Transcript

38 paragraphs

This is an automatic transcript of the recording above. It is published in full and unedited, apart from correcting names the recogniser reliably mishears. It will contain mistakes.

00:10Next.js Conf, what's up? Great to be here with you all talking about the web, the future, Next.js, Vercel. Woo. I'm here for it is such an honor and privilege to be a part of this and talk to you. So I'm here today to talk to you about some fun stuff, but before we get ahead of ourselves, lemme do a quick introduction. My name is Tejas, that's pronounced like advantageous, for those of you who may struggle to say it, it is definitely not Tejas, or you know, anyway, but whether or not I'm advantageous, we'll find out at the end of the the talk. And I'm the director of developer relations at a company called Zeta. Zeta is a serverless database focused on developer experience. Like we're trying to give like, you know, high premium Vercel, quality DX to databases, data infrastructure, search engine, the data, the whole data layer, right?

01:05But you know, that's not what the talk is about. The talk today is about enabling data at massive scale with Next.js enabling data at massive scale with Next.js. And you know, that may sound very buzz wordy and what have you, if you're judging me, don't, Anyway, no, but it's, so what does this mean? And let's get into that to start the talk with what do we even mean by like massive scalable or, or massive scale? You see, this is, this is something we need to identify to, to begin the presentation. So here, here's what I posit, here's what I propose. When you start a product or when you work in a team, usually you find yourself, for the intents and purposes of this talk. At one of three possible areas of scale with your product or your company or what have you.

01:59You may have seed scale that is you're just kind of initial, you're getting started. You maybe don't have users, you're in beta, you're just, you know, trying things out. You may be in startup scale, this is where you have users, maybe a few hundred or a thousand and people are using your thing. You've got some tech infrastructure going, it's working. And you're not nervous about getting paged at like 3:00 AM right? Or you may be at substantial scale. So you may be at enterprise to your, you know, lots of users. You might be one of the, the fangs of the world, right? Facebook, Amazon, Apple, Netflix, Google, Microsoft, Spotify, whatever. You might be either at seed or at startup or at substantial.

02:45And so when we say massive scale in this presentation, we mean like, you know, you're good past Spotify size, right? How do we get there? How do we enable apps there? How does React to Next.js and Vercel and Zeta, how do we play in that arena? Because candidly speaking, I am immensely grateful for the value that Vercel and Next.js provide for getting started. Like, it's, it's phenomenal. I mean, even before Vercel, right? If you remember Zeit, when you type in one command, now enter, boom, your stuff's live. What? How? with Next.js NPM install, React React dom next create pages/index.tsx boom, you have a web page. What? So the getting started experience is phenomenal. And then what Next.js and Vercel will do is follow you from seed scale to startup scale.

03:40There's a ton of startups built on Vercel and Next.js working to this day, Zeta is actually one of them. So that's phenomenal. And then, you know what Vercel and Next.js will do is, also follow you through to substantial scale if you want. As far as I remember, Kendrick Lamar's website is is on Vercel. I know Apple has some stuff on I think Next.js. So like it'll follow you through to substantial scale. So now how do we do that? How do we enable that for data, right? And so that's kind of the gist of the talk. Massive scale Jams, Stack, zero friction developer experience. Let's go. One of the claims I wanna make is that Jam Stack apps usually have similar functionality. Jam Stack apps usually have similar functionality.

04:33Now consider with me three apps. You may consider them Jam Stack apps because they're quite literally built with JavaScript, APIs and markup. Or you may not. Either way, they're applications that use or are built on the web, consider with me Wordle, consider with me Tinder, consider with me Uber. Okay? So you've got a game, a dating app and a ride sharing like taxi app, okay? Built with similar technology. Like if you zoom out, there's a lot that's the same, especially there's multiple cross-cutting concerns. And so what we want at Zeta, what I want personally is to have a lot of these cross-cutting concerns handled. Why? Because across these different companies you have a bunch of teams reinventing wheels.

05:24Often that is of course, until there's a standard way to deploy or a standard React framework, we keep reinventing the wheel. So what I wanna do on this talk is identify what are these crosscutting concerns before we can then talk about how you know they're solved. So let's do that. What are the concerns? So Next.js and Vercel, I don't know if I'm allowed to keep the logos rotated like this, but there's a point to it, I promise. Next.js and Vercel solve some of these problems. So cross-cutting concern in these apps in many apps, number one is, is routing, right? You have the concern of routing, which Next.js by having opinions will solve pages/filename is your route. Public/staticasset is the path to your static asset on an http environment.

06:20So, by having opinions, Next.js gives you the solution to routing in a standardized way. And tons of companies benefit number two, Vercel as a platform solves the cross-cutting concern that is deployment. Like you just run a command Vercel, boom, it's live, connect a git repository, boom, it's live. So the concerns of routing that pretty much every app has or deployment that how do you have an app if you don't deploy it? Those through these tools are solved well. I think there's an outlier, right? And this outlier is the data. Is the data. And so where is, where is my solution to not have to reinvent? And I don't mean database, right? I mean, where is my solution to not have to reinvent the wheel for data infrastructure as a whole?

07:17And I'm talking, you know, database, Redis, search engine, OLA, analytics. Where is that? That is, that is missing. But before we dive in, let's talk about these first two concerns of routing and deployment. And then we'll move to the data piece. What I wanna do is talk specifically about routing. Routing. You know, React is a beautiful tool, right? React is a, is a library with like protection against scope creep, right? React will not ship its own router. React will not ship its own data layer. React is like, no, no, no. Listen, we make building UIs good peer hit. If you want to fetch data, SWR, React query, what have you, the ecosystem will help you. But React has a very nice limited scope, which is actually beautiful cuz it can do its job well without doing too much.

08:13Thankfully this gives way to frameworks like, like Next.js remix or even libraries like React router for example. What I wanna do to understand the complexity of routing, look, it's a talk about developer education. Okay? So let's just educate our, so let's see if we wanted to build our own router, let's just see what that would take. And then just marvel at, at how Next.js and these solutions save us. So what we're gonna do is we're gonna switch to the laptop here. You'll see my presentation, whatever. It's fine, yes, this is my slide. But let's move to the editor. And what I wanna do is let's create like a, if I can call it this, a poor man's router. Okay? So what I'll do is npm init. This is literally like, look, there's nothing here.

08:58It's an empty project. So we'll npm init and we we'll npm install, I don't know know like let's React React dumb parcel TypeScript. Like I'm just installing everything. Dev dependencies, dependencies, whatever. I don't care. Types React types, React dom. This should be good. Running npm install in a conference talk, on questionable wifi. Is he doing that? Yes he is. Anyway, I think, I think it'll be fine. Let's come back here and look. It should be done any minute now. Is it done? Let's take a look-see. Nope, it's not but hey, it's, it's coming. In fact, if I could time travel and speed up this video, I would, but it's fine. This is gonna take like just a couple seconds more and we are good.

09:54Awesome. So we have some stuff, let's go to work. So now let's take a look. We have package and packaged lock. Let's get started. We'll create a two files index... Let's bump that font size index.html serve as our root. We can even just get started with some code in here. This one, let's give it a title. My poor poro man's router and we'll add a script tag index.tsx type is module and we'll add a div id route to tell React where to hydrate. Okay, perfect. Now of course we need an index.tsx. So what I'll do is index.tsx and let's get to work. So let's just do everything in one file because we're building a poor man thing anyway, we'll do is import create root from React dom/client. We'll say root is create exact wow, copilot.

10:55Awesome. All right. And now we'll route dot Render app, which, what is that? So let's create that. Let's just for the sake of making sure everything's working, just do a hello Next.js. Conf. All right, so we let's, it's giving us red underlines cuz we don't have React. This is fine. Okay, save this looks good, Index looks good. Let's try and run this with npx parcel index.html. Okay, 400 milliseconds. What do we have? Great, it works. But we're building a poor man's router here. So what does it take to have a router? We would need a way of knowing like the path we're at. And then if the path matches show stuff, this is a poor man's. We just wanna understand the complexity of reinventing the wheel here.

11:57There's a point, I promise. So we come back to the laptop, Hello Next.js conf, let's create some routes. So what we wanna do is add some links to start with. So we'll do a href / about, sure / Twitter, maybe do this. And then we will add, we need a route component. All right, let's do that. So const route, it's a component. In fact it's a function component. Ooh, what was that come back function component that has props with children and it really just needs a path, with a string prop. We're using TypeScript here. I hope you don't mind, if you do mind, I can't help you. So what, what do we wanna do here? If path equals equals equals current path, then return children, right? And children will come from props, otherwise return, null.

13:02So if the path matches whatever the current path is, we show it. The problem is, ooh, this looks that this looks pretty bad, huh? Let's maybe fix that by using Semantec HTML to the rescue. Just do some allies, some ULs, Oh that's not an element. ULs here. Boom. Much better. Okay, so this is, we don't know what current path is. And also it's declared but never read, which is, which is fine cuz this is a bit nonsensical. So we need a way to get the current path. The current path is considered context. So we'd have to use some type of context object. So let's create that. Let's do const router context equals create context. And we'll say, you know, this literally has the current path, which is the empty string by default.

13:59Cool. So now you can say current path is use context, router context, boom. All right, so far so good. I like it. Except the router context. Our app here is not given the context, it's not wrapped in the provider. So let's create a provider, router provider. And what it is, is a router context.provider that wraps children. This is a function component that has props with children. Nothing else, okay? And so we get children here, awesome. But now you can see it wants a current path. So current path, we can also give it a prop, maybe current or initial path string. And then we'll get the initial path in props. And value is current path is, it's not initial, nevermind. We'll make this from the props.

15:09Okay. And then we'll just destructure. Perfect. All right, this looks good. So now we have a provider to provide context. Let's wrap our app with it. Just like that you notice there's a prop missing. So we're client side rendering. So we can easily say that we get the current path from window.location.pathname. Okay good. So everything's wired up. We're just not using the routes and it doesn't know what path is, which is a prop. I should maybe get it from here. Okay. So let's use this route component now. What we want is let's go to section and we'll do route path. If I'm on about, then show these things. Lorom, maybe I can do something like this. And for another route, if I'm on Twitter, I can show like, you know, follow me.

16:11(laughs) Okay, so does it work so far? Yes. We're not on a path. So we see nothing. But if we go to about, okay, and then Twitter. Okay, so we built a router in one file. That kind of works. Does it work on here? Does it work on full page loads? Let's, so it does, The problem though is in my markup, there's absolutely nothing. And that is because this is rendered fully on the client side. Now if we wanted to Render it on the Server side, we can add, we can add some stuff. So we can, instead of giving the current path here, we can actually create another file, call it server.tsx. Lets just npm install express here as well. And now, yeah, you're on next dev. (laughs) I'm so used to Next.js npx parcel index ratio.

17:15So now what I'll do is I'll import express from express start my express app like this and I'll app.get path, right? And I have request response. And what I'll do is path is rec.params. Sure, I can actually just de structure, let's do this. Oops. And now what I can do is Render to pipeable. Let's just use use TypeScript from React dom Server Render to pipeable what? Render to pipeable stream like this. We'll Render our app. Exactly the same app as here to pipeable stream. And then we'll just pipe it to res, So this is how we'd server render. Of course we'd export these things. So I'd export a router provider I'D export app. It's convenient that everything's in one file. Don't do this in production.

18:20Router provider app, right? Is that missing some type of prop, I need React. This is just a, you don't actually need React if you have Next.js configured properly. But anyway, so this is fine, this works. And instead of Windows, since I don't have it, I can do like req.path and boom, I have Server rendering, right? So yes, this is good, but will this see me at seed scale? Will this work at seed scale? Sure, this is kind of seed scale kind of works. Will this work at startup scale? Look, I have, I have client rendering, I have Server rendering. It will work to some degree at startup scale. But what if I start accumulating users? What if I start getting a bunch of users? I'm scared of PagerDuty with this.

19:12No doubt it'll work. But should I? I, I'm not so sure. Okay, will it work at substantial scale? Absolutely not. Shut the front door, send it out, it's not gonna work. This, this is, this will break. Especially as you onboard a bunch of engineers, you grow. You're like, why? Why aren't you using Next.js? No. So this kind of approach appreciable at seed scale, kind of okay at startup scale, not good at massive scale. What's the alternative? Coming back to the topic of routing. The alternative is of course to rely on, to stand on the shoulders of giants, to lean on tools. Next.js, open source tools, remix, React router. That's why they exist. And with Next.js and Vercel, it's a solved problem at all three levels of scale.

19:58Vercel specifically Next.js even has this layouts RFC that's gonna take it to the next level. There's a lot happening at that level of scale. I hope this has given you really, the reason I even did this little exercise is to give you a little window into the complexity and the edge cases that exist that if some other team has their full-time job dealing with, then we don't have to. Okay, moving on. Let's talk about deployment. I mean, as the second pillar, like if we kind of go back to the, the butterfly analogy here, right? Deployment, I feel like, look, it's, it's a conference organized by Vercel. Like why do I, I feel like I don't even need to talk about this, but for posterity I will. This crosscutting concern across Uber, and Tinder, and Wordle handled at Vercel, does it work at seed scale?

20:47Yes. Yes it does. You can run a command Vercel seed scale, you're good. Does it work at startup scale? Zeta is evidence that it does. Does it work at substantial scale? Yes. In fact, how it works, if we look a little bit under the hood, part of it is this awesome offering with Vercel's edge, right? Vercel has a content delivery network where your code can be served from geographically a lot closer to users, therefore they get their stuff faster. The geographical distance light needs to travel, the internet travels across a speed of light. If you didn't know that now you know, but speed of light is a real limit. So the shorter the speed of light distance is, the faster people are gonna get their stuff, the happier they're gonna be.

21:29I feel like with Next.js and Vercel deployment is a solved problem. Connect to git repository, forget about it. You can tweet that, that then leaves, right? The third pillar, we talked about data. Data and this is important, you know why it's important? Cuz we just implemented in the front end a router, we deployed it. So we have all this front end power, right? But what happens when the Backend is needed? Which of course you need a Backend for apps like Wordle stores the answers of all the words in your browser. Some would say they're using the browser as a Backend store. I don't know if I agree with that. But typically if you're building something like Wordle, you wanna keep the word secure in some foreign system, you'd need a Backend.

22:16Uber needs a Backend, Tinder needs a Backend. What is being done for data? What is being done to facilitate true Full Stack deployment? You see, because Vercel and next years are awesome where the front end is concerned. But what do we need to go full Full Stack. And you know what I mean by full Full Stack here, I, I mean there's a ton of developers these days that are really great on the front end and kind of lacking on the Backend or great on the Backend and lacking on the, So Full Stack is a loaded term because here this is a slide from Swyx at React Miami and this is his take on a Full Stack developer. Here, let me zoom in. That that here, (laughs) that's like, yeah, so that's kind of what we see with a Full Stack developer.

23:02So if we come back to this, this butterfly routing deployment and data, this is where Zeta fits in. A lot of people call Zeta like the Next.js of databases or the Vercel of database. That's kind of what we wanna be the, the Backend that feels as nice and premium to use is Next.js and Versal. In fact, I'd like to show you that right now. But again, the goal is not adoption. I don't really care if you use it. I'm not trying to sell you anything. I wanna show you what it is and then talk through how we build it using Next.js to kind of give you an appreciation and an idea for the problem we're solving and how we solve it for education. So you can grow in your career and do all the things. Okay, so let's get into this demo.

23:50If we come to the laptop, again, this is what Zeta looks like. It says launch your first workspace. A workspace is kind of like a team or an organization in GitHub. So if we add a workspace, I don't know, what should I call my team? I'll call it Tim Neutken's fan club. (laughs) And so now I have this, and so in Zeta I wanna create a database. So add a database, I would call it stuff. So I'm creating this database and just like instantly the database exists, that's really all I need. I wanna add a table to it. Zeta is schema full. So I'll create a data, a table here and I'll call it a demo database. So Zeta will help me kind of understand what's needed or how it works. So not created for me a demo database.

24:39I have two tables, posts and users. I can explore the schema here and kind of look at how they connect to each other. I can see that author and posts is connected to users. Great. And I can look, this actually looks pretty nice. This is a database, believe it or not. And so what I can do is I can maybe say, you know what, I don't want any of these columns except I just want the title. This looks good to me. So now I wanna query this thing and use it in my Next.js trademark app. So I will go over here to get code snippet and I'm just gonna give myself some credentials. So I'll say yes and I'll copy the snippet and literally from my own browser, I can go here, I can paste and I get back the data exactly as I see it here I get an ID and the titles, genderqueer IB male, genderqueer IB.

25:30I get back exactly what I'd expect. This is pretty nice because the time from provision to query is pretty virtually nothing. And now I can take this back end, this database, use it in my Next.js app and I'm ready to go. But why even stop there? Like look, I can, I even get a playground where I can play with TypeScript. Literally I can like run this, it uses real live production data but you know, whatever. And I get back the data. In fact it's type safe. So if I have a like a user's table, it knows so I can't like mess up this or it's gonna yell at me. What? You don't have that table for example, it goes even more than this. Like I could use the CLI to zeta in it and it will ask me like what database do you want stuff I can generate TypeScript code.

26:34So I can query it in a type safe way and it will pull down a JavaScript client or TypeScript client in this case for me to use. So that thing that I did in the playground, just copy paste snippets into my editor and it'll work. We do support the notion of branches of your database, just like git, you can open pull requests and everything. And lastly what I wanna show you is I can literally like from the CLI edit my schema and I can like add another table from here. So I can just add a table Next.js, hello. And in my table I can add a column. The column name is tejas, it's a string. I don't know why enter. And when all of this is done, I will run this migration. Yes, it's creating a table and it exists, right?

27:26That's the developer experience we're going for. And what I can even do is run a command zeta random data. And so I can do random data table is next, what is it called? Next.js. And it will just like insert boom, I have data. I can once again query this with temporary credentials either in my console and curl whatever, and I get back stuff. So that's kind of what we're doing and that's the developer experience we're trying to create to mimic and really complete the ecosystem of powerful jam stack tooling. But I, that's maybe an impressive demo and you wanna check it out, cool. But this is about your education and about how we're doing this to help you understand what a lot of these companies do.

28:12And I'd like to show you that here in our time together. So if we come back here, Excalidraw, wonderful tool. So considering again Uber, Tinder and Wordle, maybe not Wordle, but any company with a database, right? The wheel is reinvented a lot kind of like routing and React before Next.js kind of like deployment before you know Vercel or Zeit, the wheel keeps being reinvented. What wheel? This wheel. So you start with a database, why is this line so dark? It's really like not, let's, let's try this again. So you start with a UI, actually some type of UI deploy with Vercel talks to an API, maybe a Next.js API route, right underneath is a database. And so this sometimes will do the job, but oftentimes what you will need is you'll probably need to distribute your API to load balance.

29:11And as data demands add up, you'll wanna scale your database maybe at first vertically, but at some point that will run out of steam. And then you'll need to deploy horizontally. This is already getting really complex cuz you need to deal with primary replica consistency and so on. But then as you grow, you're gonna collect a lot of data volume and when you have a lot of data volume, you need a way to find it. Can you imagine Instagram without a search engine? So you're gonna need a search engine. This makes up data infrastructure. So we'll add here, search. Now you have the complexity of like how does the database replicate into the search engine? How does my app talk to the search engine?

29:54Is should we distribute the search engine? And you'll think, okay, that's great, that works, but it's still slow. This will not make it to substantial scale happily. Why? Cuz databases often read from and write to disk what is faster than disk. Exactly. So you'll need another piece of data infrastructure, you'll need some type of in memory store, maybe a Redis or something like this. And then you know, you'll have your API to talk to that first. And if it's a cache miss, then go to the database. It's a whole story. But then you want visibility into what's happening here as well. So you'd need some type of like online analytics processor. It's probably distributed as well. And all of these things need to feed into it.

30:41Oh my word. And of course you'd need some type of, whew, you need some type of dash boarding tool, maybe a Grafana or something. And so look, look, check, check, check this out, this is your, whoa, that's a very confused line. Anyway, so this wheel keeps being reinvented like literally. And, we like the example of Next.js. It's like, hey, we have opinions about routing. Use it. Your life gets easier. You can actually build Tinder faster without implementing routing. You can build uber faster without implementing routing. Not that they use Next.js by the way, I just keep coming back to these companies with Vercel, right? Hey, you don't need to roll your own deployment, we'll do it for you. And with Zeta data infrastructure, we got you.

31:32If you wanna build a dating app, if you wanna build a ride share app, we got you. So that's kind of the, the gist. And this is, this is here, look again, this is the underlying way of how it's built. One implementation detail is we use Kafka to replicate the DB inter search as a queue. So actually this is more, if we can squeeze Kafka in there, this is more indicative of of our infra under the hood that we kind of rent out to people. But the guarantee is you don't need to pay a team of DBAs, you just kind of use a free product or pay as you go or whatever. So it saves complexity in exchange for convenience. Let's wrap up this presentation. I'm a little bit over time, I apologize. Let's go to the presentation.

32:17So we just finished up the demo, Let's wrap things up. So we looked at how it's built and what we're learning from Next.js. Let's talk takeaways take away number one, wheels don't need reinvention, right? I feel like we should be good to stand on the shoulders of giants and actually build a product we wanna build instead of keep having redundant groundhog Day discussions. Next.js helps this. Vercel helps this. With Zeta we're aiming to help this. Number two product focuses the goal. If you spend your time focusing on what database to choose, what deployment platform to use, what React router to use, you kind of lose sight of your product. For us, the goal is to enable people to focus on the products.

33:05And lastly, we believe that opinions facilitate productivity. Next.js allows us to move so fast because routing just, you know, you have, what is it? You have a directory pages slash something good. That opinion allows us to be productive at Zeta as an abstraction layer on databases. We try to have the strong opinions so that people can just follow a pattern and build the apps they want. And with that I wanna say, hey, thank you so much for, for having me. That Twitter handle is outdated, it should say @zeta, but it's fine, you'll figure it out. I appreciate your time and attention Next.js conf, it's been real. I'll catch you in the next one. All right, peace.

More talks

All 19 transcripts

Elsewhere

There is every talk I have given, all 69 of them, ConTejas Code, the podcast, and Fluent React, the O'Reilly book on how React works inside.