How to build a Multi-Region Application on CockroachCloud

– Welcome to today’s webinar everyone My name is Meagan I am on the product marketing team here at Cockroach Labs, and I am here with Lakshmi and David so I’m going to let them say a few words to introduce themselves – Hi everyone, my name is Lakshmi I work on the product team here at Cockroach Labs and I’m the general manager for our cloud offering which is CockroachCloud – David, introduce yourself – Hello, my name is David I am a cloud developer I am based in Spain, and I wanted to say thank you to Lakshmi and Meagan and the Cockroach team for inviting me today to join their webinar It’s great to be here – Perfect Before I pass it off to Lakshmi and David to talk a little bit about David’s experience using CockroachCloud, I just wanted to do a few little pieces of housekeeping So the first one is you’ll see a Q&A panel at the bottom of the screen Definitely do ask questions in that panel throughout the webinar, so if any questions arise we’ll be keeping an eye on that and addressing them And then at the end when the webinar ends, you’ll see a survey and please fill that out It’s really helpful for us to hear your feedback for all of our future webinars And then finally we’ll be sending out a recording after the event so everyone can can watch it, watch it again, watch it as many times as you want So that’s pretty much it Thank you everyone for being here today And now I’m just going to pass it off to Lakshmi and David – Great, sounds good Thanks again for joining everyone Good morning, good afternoon, good evening, wherever you’re based Like we said a little bit earlier, David was one of our first customer, earliest early adopters of CockroachDB So they’ve been working with CockroachDB for a few years now And when we released our cloud offering, they were of course one of our first customers for CockroachCloud So I’m very excited and grateful that David is making time in his busy schedule to share his experience with us So David, can you share a little bit about your team and the kind of work you do at your company and what your day looks like? – Okay As I said before I am a backend developer I work for an electronic access control system manufacturer, and I belong to the cloud team And basically we are developing cloud based solutions for the company And I think that our technology stack, it’s pretty similar than the one you are using in CockroachDB We’re using Go, Docker, Kubernetes, Google Cloud Platform services, following a microservice philosophy – Great, sounds good I remember when I was talking to someone on your team early on, the fact that we used very similar stack was one of the things he kept in mind, like the fact that we also use Go and your team uses Go and Kubernetes and Docker That was one of the criteria that made the team feel a little bit better about using CockroachDB Great What kind of applications do you have today that run on CockroachDB? – Basically all the cloud applications we are developing in the cloud team in my company work with CockroachDB But we have also another microservices that are also working with CockroachDB For example, our distributed locking application, our dynamic discovery service and so on But sometimes you have to also use the third party libraries that might need to store to use a database to function For example, we are using an OpenID Connect provider that used to have only support for MySQL and PostgreSQL and following this philosophy of only using a single database, so we have only to use and to maintain a single database We are the support for this OpenID Connect provider And since then we knew that more companies are using the same combo, CockroachDB We use these libraries so we are very happy about this – Yeah, I remember that was one of your early requests and we’re grateful that you did all the work to make CockroachDB work with OpenID It’s exciting to hear that few others from that community are also using it What is your current CockroachCloud setup? Like I said, you are I think the first 10 customers who ended up using CockroachCloud and gave a lot of feedback on what should the console look like, what should that process look like So yeah, what does your setup look like? I think our audience would be particularly interested in the multi-region setup – Yes We have three regions One in Asia, another one in Europe, and another one in the US We have three nodes per region So we have nine nodes in total

And using this global Cockroach setup, is actually helpful for us We can take and form some CockroachDB special features for this Like follower reads, geo-partitioning, automatic follow-the-sun, and our customers this way can have the same experience wherever they are – And you run in GCP, is that correct? You are in Google? – Yes, we are also in GCP – Thank you How did you determine that you needed a multi-region database for your application? Because it’s not always obvious that you need one So how did you come to that decision? – Well, our customers are spread all over the world and we needed to give them the same feature set and the same experience wherever they are We wanted to give them the same experience And also our customers might be also spread all over the world So with CockroachDB we can give them the freedom to choose where they want to store the data For example, to comply with some data regulations as GDPR in Europe and all these kinds of regulations that you have to have your data stored in Europe for Europe customers and so on – Yeah I think that’s a pretty common pattern that we hear from customers like GDPR, especially for European customers is huge Right, you want to keep data to a specific locality and then increasingly most companies do have customers kind of everywhere in the world and therefore it’s Yeah, sorry I’m getting distracted by a Q&A question here But just to finish that thought, yeah, just to provide the same level of latency to all of the different customers around the world I’m going to check on a Q&A that’s directly related to the question We have a question from Miguel The question is data residency, so keeping data close to a particular geography, locality, like what you were just saying Is data residency an issue for you? Yeah Can you speak a little bit about that? – Yes Because all of our customers are spread all over the world For example people that are located in Europe, might need to keep the data only in Europe to comply with some regulations, say GDPR or maybe some people from the US don’t want their data outside the US for whatever quality issue or I don’t know So we added partitioning there Yes – Yeah And you specifically use your partitioning to keep the data in specific localities? – What, sorry? – What is the feature that you use specifically to keep the data in one region? – Geo-partitioning Yes – Right, yeah We have another follow up question from Frankie The question he was saying, I didn’t understand the three to four features that David mentioned regarding multi-region capabilities like geo-partitioning Could you repeat the capabilities? So I think you were talking about the specific features that you use for multi-region So follower reads, can you just repeat that list? – Yes More or less the overview is that with CockroachDB you can choose where, you can store your own data But you can also give your customers the ability, so they choose where they want to store their data also So it’s very good for our use case – And you had also mentioned follow reads and CDC which we’ll talk a little bit about that But those are the two other features that he had mentioned, that he specifically talks about And we’ll get to CDC in a bit – Yes For example with the CDC, we try to decouple some requests from the request spot, from the hotspot and give our users everything okay Basically, do the heavy work on the background and be eventually consistent But we are using CDC for much more things For example, we are replicating read only data as materialized views for better performance

and to keep our queries simple, easy to understand, easy to change, avoiding multiple joins, and avoiding complexity at all At the end we are not database administrators We are developers So we wanted to avoid all this complexity We are introducing also for cache eviction to update or to do delete data synchronously So it’s a huge blue sphere for us – Yeah So just to summarize, to specifically answer that question, it sounds like David’s team uses geo-partitioning, CDC and follower reads, which are the three specific features that he uses for this multi-region application, the distributed locking service – Mm-hmm – Well, we kind of just specifically touched on this Are there any more questions specifically on this? All right, we’ll keep going Thanks David Specifically on CDC, I think you have interesting story here because you talked about how you actually use the core version of CDC even though on CockroachCloud you do get all the enterprise features So you can use the enterprise version of CDC, but for your team, the core version actually made more sense So can you speak about how you use that for CDC? – Yes We felt that the enterprise was very tight with the Apache Kafka, and this is okay But we felt that it was a very heavy and complicated for our use case, as we don’t need a strict ordering and we don’t need also exact once delivery So the core option, they pass more freedom to use whatever messaging broker we want to use, for example, GCP Pub/Sub and that’s our choice – Did you say GCP Pub/Sub is what you ended up using? – Yes It’s the way I’m using GCP Pub/Sub along with the CDC core – Got it It seems like a question around that? Got it Yeah, we’ll get to that in a minute Can you speak a little bit about what is the hardest part of building a multi-region application so far? Because this is something you’ve been working on for about a year now, maybe more – Yes, yes Well, having data and users spread all over the world, it’s pretty hard by itself And Cockroach can help with some of the future as I mentioned before But also you have to deal sometimes with third party libraries that might be compatible But if you want to use CockroachDB syntactic SQL for multi-region capabilities This might not work So you’ll have to build with it or try to interpret or to attract that tool to CockroachDB That is what we did For example with this OpenID Connect client – One of the questions is when you We talked about how you have nine nodes in three different regions One question is what is the size of the data, size of the database? How many queries or transactions per second? How big is your kind of workload? – To be honest, I don’t have the numbers I am not sure that I could give this kind of information, so I have to skip this, sorry – That’s totally fair That makes sense – We can follow up later – Yep We’ll follow up on that a little bit later I’m curious, as somebody who’s built multi-region applications, if there’s any advice you want to share with developers who want to build a global application like yours? – Yes As I said before, building a global application is very hard You have to think about a lot of things: about the latency, about the customer location, data synchronization, networking, data regulations and many more So CockroachDB, it’s a very helpful tool for us and specifically the cloud version Because we can focus on our actual work as developers

to develop and not having to worry about the operations, database administrations, laws and all these kinds of things – What would you say is the hardest part of building this application? – More or less what I said before, having to manage a lot of data and this kind of database, these clusters So this is the reason why we choose CockroachCloud to avoid this kind of problem and focus on our daily work – Great I think there are actually a number of questions from the Q&A, so I’m just going to go through them as I sort through them Bear with me for a minute So one question that we have here is maybe for later, but we would love to hear about deployment options Is there a Terraform operator? I think that’s a question for me so I can answer that – [David] Okay – So we don’t have a Terraform operator today for CockroachCloud, but that is on the roadmap and I’m happy to follow up on that Can we get the emails of the people who ask the questions? – Yeah – Great, okay So Miguel, I’ll follow up on that It’s something that people have requested, so I’m curious and interested to learn a little bit more about what specifically that you’re trying to solve with it One of our goals for this year is definitely to expose the APIs that would allow you to create the CockroachCloud clusters through this UI So rather than going in through the UI and actually putting the nodes and the cloud provider and so on, just being able to call the API So internally we are working on that and then we expect that to be kind of customer facing sometime this year, broadly speaking But the usual caveat that roadmaps change and so on So that might help a little bit about your deployment question, but I’ll follow up on that after this webinar Great And then there’s another question Can you scroll this down? Okay How do I, oh, there we go Frankie was asking, are there specific hotspots in the data access and how you dealt with them David, are you able to answer that? The question about hotspots Just for context, this would be case where the same rows or the same tables in the cluster gets repeated so it creates a hotspot – You kind of have some hotspot For example, when a lot of people, it’s trying to read the same data in some place and this is covered by CockroachDB with this follow-the-sun and follow-the-workload feature So they kind of move the withholder closer to the users So the data access is much more faster compared to another database if you have to go to the US and back, for example, to have a reset – Mm-hmm Yeah, that makes sense It sounds like follower reads does solve that specific hotspot problem for you, your case, yeah And interestingly on the topic of hotspots, one of the feedbacks for our web UI team, which is the dashboard where you can see kind of your performance and latency and various other metrics, is being able to see when hotspots occur Just having that observability to see it as an important piece of it And I know that there’s a separate team that’s been specifically working on that So I’m having follow up on that Frankie, if you’re more interested in learning about that specific topic Another question, I think we kind of touched on this, but it’s around, what are the limitations or bottlenecks you faced? I think David kind of talked about some of the challenges, but I don’t know if there’s anything else you’d like to add – Not really I think that with the choice that we have right now, we didn’t experience any bottlenecks yet But maybe we could in the future Nothing to mention right now – Yeah One of the things you mentioned earlier on, in case people didn’t hear, is when you do start on an application like this, the various third party tools that you work on in making sure that integrates and works well with your cluster and database is an important piece And you David spoke specifically about the Open Connect and Hydra? Right? – Yes, Hydra – Hydra, yeah., great We have another question that is,

how do you debug your deployment? – Okay First off all we have a lot of tests before deploying We are using a Data-dog tool, it’s called go-txdb and this– – Can you repeat that one time, what’s the name? – Data-dog tool, it’s called go-txdb – Okay – And this is basically a tool that let’s you make some queries in tests without modifying your database And it’s a very great tool for us So we’ve had a lot of tests and once the code is deployed, we have a lot of traces, coding traces We are using OpenTelemetry for that – And you also have a staging environment and a production environment, right You have two separate instances, is that correct? – Yes, we have kind of a deployment It’s not a stage in production because that kind of intermediate deployment, it’s something we also use, locally in our company So we are kind of testing it before going to real one – Yeah Yeah, that makes sense Great Are there any more questions that we haven’t touched on? Curious Okay David, I don’t know if you are familiar with this, it might have been before your team, before you specifically worked on this application, but what other databases were What was the evaluation process like? Did you look at other solutions before you selected CockroachDB, and what were those and why? I’m always curious how people end up with the product – Okay As we are using Google cloud services, the nature of it was to use workload Cloud Spanner, but we felt that CockroachDB was better for our use case because with the workload Cloud Spanner you only have the option to deploy it in Google cloud So you are kind of locked With CockroachDB, you can deploy it wherever you want I’m not sure CockroachDB is open source It’s working in Go, so we can look to the Go and even fix bugs if we find them And also it’s using the Postgres wire protocol and the Postgres syntax, so they are kind of trying to not reinvent the wheel here, which is a huge plus for us We don’t have to learn new syntax to use CockroachDB – Mm-hmm, great How big is your team? How many people on a one day basis? – I am not sure I could share that information, sorry – Okay, that’s fair That’s totally fair No, understood Are there any more questions? Are we good? – No questions – Okay, cool I think we’ve answered most of the questions I’m going to follow up with Miguel about the Terraform operator and was there one more followup? I think that was the main one that I was going to follow up on Great Thank you everyone for joining David, thank you so much for making time on a weekday afternoon, your side to share your experience, yeah – It’s been a pleasure to be here Thank you – Yeah, thank you so much David That was awesome – Cool – So there’s a few resources that you can definitely check out There’s one that’s for building a multi-region app on CockroachCloud that is in our docs and we can send out the link after this, and that’s a super helpful tutorial And I think that’s pretty much it That’s pretty much all we had I just wanted to do a really quick plug for this upcoming live stream we have on Twitch where we’re experimenting with Twitch Now we’re going to do a coding livestream and anyone can join So that’s on March 13th and it’s also about building a multi-region app So that could be relevant for some of you guys who have joined today And then if you want to check out CockroachCloud, you can just get started on our website, cockroachlabs.com/cockroachcloud And I think that’s pretty much it – Yeah And if there are any follow up questions, we have a community Slack channel now, so that’s a great place to just post questions as you’re trying out the product and have any questions

Customers like David who’ve actually used the product are happy to chime in or obviously any of our kind of engineering employees within the company also help answer the questions – Yeah, absolutely that’s a good call – Great, all right – Cool Well, thank you everyone for joining and have a great rest of your day