Hello and welcome back to another episode of Insta Blinks. Last episode we talked OpenSearch 3.0. It was a great episode. I wanted to get more in. We didn't have the time, so I have, um, Andrew and Sara back again to talk more. OpenSearch 3.0. Welcome back gentlemen. How are you doing? Good. Thanks for having us again. Yeah. So great.talk. Last time we talked about what's coming, we talked about some of the breaking changes. We talked about what to expect. We talked about, um, Lucene ten major milestones. So much good stuff. If you haven't watched this episode, please go back and watch the previous episode. Today I wanted to focus more on kind of what to expect now that we've talked about the cool, shiny features, what does it take to actually move to 3.0 and what does that look like? If you feel like you missed the boat, you're still on Elasticsearch and you finally want to move over to OpenSearch 3.0. What does that look like? So we're going to jump into a few of those things here in this episode. And so we're just going to get kicked off and go before we get into the migrations, I want to talk a little bit about how some of the features that we talked about last episode came to be. So this has been run and owned by AWS, right, for a number of years. Last September, this was put into the OpenSearch Software Foundation, which is a part of the Linux Foundation. How do we see the features that are being released here in 3.0 being affected by either contributors orcustomers? Right. So I think OpenSearch has always branded itself like a community driven project. And with 3.0, I think that we see a lot in practice, like even the proposal for 3.0 was you know, discussed and opened on GitHub. It was, uh, it came along with the community feedback, not just from the timeline, but also about what we are expecting in terms of the core dependency updates, what we are expecting in terms of the breaking changes, what we are expecting in terms of the roadmaps. So OpenSearch does have a public roadmap, and that roadmap is maintained by community where people can go and ask the right question, hey, is this feature coming in like 3.0? Is this something you know you would want to delay the timeline for? So we have had those conversations as well. And then we I think a lot of community input came in from the roadmap perspective like features and also the timeline. So the whole inception of 3.0, like from the idea to kind of the journey where we are right now in the process of getting beta out has been pretty much, uh, coming from the community standpoint. We talked about how in the last episode, Uber had some functionality that they're adding to this, which is really cool. Um, and it's great to see that. But then also, I feel like maybe there are individuals, um, in the community who are like, well, yeah, of course Uber is going to be the one to help. Like, they're always doing these cool things, you know, is it possible for individuals to contribute or smaller companies to contribute to this, or is it kind of just for thebig boys? Whatdoes that look like in terms of the open source, the open search features and functionality? Right. I think it's not limited just to the partners or the bigger companies who work in or who have adopted OpenSearch as a project that they want to contribute on. Uh, a lot of these contributions are coming from individual contributors, or I would say developers from different projects, and especially that is very much seen during our community meetings, because whenever we kind of we do kind of run community meetings for different areas such as search, indexing and stuff like that, and then they are kind of scheduled and publicly broadcasted on meetup. So we do see a lot of these individual contributors coming in and joining and talking about some of the issues. Good first issue that they are looking to kind of work with or even reporting bugs. And, you know, a lot of, uh, feature requests that comes from these individual contributors as well. What if a, an individual is interested in OpenSearch? Maybe they're not a developer or engineer and they want to be, you know, help contribute. Are there ways that they can contribute somehow to the project? No. I think that's a very good question and something that is being asked a lot of times when we go in conferences. So, you know, I'll just give a touch point and I'll let Andrew talk more because Andrew has been driving these, you know, collaboration with partners and also these contributors. So in terms of resources, uh, there are getting started guide in terms of how to kind of start thinking about OpenSearch in general. Like how is the architecture like what is the overall functionality suit. And then also from a developer's perspective, how do you kind of start on the GitHub, how does a build repository setup and then you know how to even make your first contribution. So we have like some dedicated, uh, I would say documentation and some videos around it as well. Uh, but yeah, I would let Andrew talk more here because from an engineering standpoint, he has interacted with a lot of people and gotten them onboarded. Sure. Um, I mean, the best place to start, as always, is just to go OpenSearch org and you'll find all the resources, uh, ultimately linked from there. Um, but as far as to answer your question of like, if you're not a developer and you want to contribute, I would say the best way to contribute is to, first of all, use the product, download the Docker image, download the tarball. Actually, you know, run,the run the software and get your hands dirty with it. You'll find lots of uh resources on OpenSearch, org and more broadly on the internet about how to do that. And then, uh, if you want to contribute, um, you know, if you, uh, if you find bugs, if you find features, come to GitHub and open the issue to discuss that. We also have the forum. You can find that linked from OpenSearch org. Um, and a public slack instance. So if anybody wants to chat on slack, you can uh, um, you can do that. And we get quite a lot of engagement on slack as well. Um, so those are the best resources. Additionally, if you go to meetup.com and search for the OpenSearch project, you'll find, um, all the various community meetings and triage meetings and things that we have scheduled there. Um, and ifthere's a particular component area or, um, you will likely find a meeting focused on search, for example, or indexing or something like that. Um, feel free to join one of those meetings. We, uh, they're open to everybody. And yeah, lots of ways to contribute. Yeah, I love that. Oh. Did you have something more? Sorry. No, I was just trying to touch upon on a couple of more avenues like we do have, uh, the forum. So open source forum is also a good platform for asking questions and getting some traction from the other maintainers, like Andrew or maybe other people who are running into a similar problem. And also we have the playground. So these playgrounds are a good way to kind of experience some of the features early on and try and see, uh, how these features would function and things like that. So a lot of dashboard features or even observability and those features are available on these playgrounds. So you can go try and create some ingestion workload or create some test data and try to get that experience. Yeah. I love hearing that there's something for everyone here. Right? If somebody wants to work ondocumentation or if they want to help with features in actual coding, they can or they can be a part of the community, whether they're lurking in the forums and in slack or contributing there. Um, and that playground, it sounds awesome. And actually I used it here about six months ago, and it was great to be able to go in and spin that up really quickly and get my hands on, you know, whatever version was,the latest at the time. Um, so for individuals that want to try this out and they don't have the resources, do check out the playground. It's great. Um, so before we go into Segway, into the migrations and kind of what that looks like, we talked a little bit about OpenSearch having, um, and GitHub having resources. But, you know, I think people as they hear about 3.0, their first questions are what's going to be in it? What does this mean to me? And then how do I move to it or how do I test it? We've talked about each of these pieces a little bit, but is there documentation available or that's going to be available with this release for people to, um, know what to expect and to have the steps to move forward? Yeah, absolutely. Um, the that information is, uh, is mostly available today. Sort of spread across various GitHub issues and is probably not theeasiest to consume. But certainly by the time of release we will have blog posts, we have release highlights, release notes. All of that will be, uh, on the on OpenSearch org in an easily consumable with uh format withall the best stuffsort of upfront as we are still in the release process now. Like, you know, lots of things are still in flux. Um, it's always an exciting time as we get closer to release. So, um, yeah, if you, uh, in your, the specific, uh, repositories or areas that you might have interest in, there will be sort of documentation about whathas changed and what's there. But yeah, definitely uh,not quite ready yet, but there will be release blogs and release notes withall the highlights will be available on the website. Awesome. All right. Big shift here. People want this. People want to put it into their environments. Um, let's talk about migrations and upgrade paths. Right. So anyone that is currently on 2.19 when this comes out, Andrew, what does that look like for them to upgrade? So a couple of options there. If you're on the latest, you know, the latest previous version which is 2.19. Um you can do in-place upgrades. You can do um, an in-place upgrade is usually you will just bring down one node, install the new software, bring it back up. It will connect to your 2.19 cluster. They still know how to talk to each other. The data format is compatible. Um, and you cango through all the nodes in your cluster and upgrade that way. Um, additionally, you can take more of a,blue green type strategy where you spin up new nodes, connect them to that cluster, move the shards from one node to the other, and then once all the data is off the old nodes, you can tear them down and you're on the new version. Um, because 3.0 maintains compatibility with that last minor, you have all those options. Um, now, if you're coming from something potentially older, uh, the three point nodes are not going to know how to talk to these older versions. And thedata format of your indexes will also not be compatible with the latest, um, the latest software either. So for however, you do have, um, migration options, I would start out by plugging we have what's called the Migration assistant for OpenSearch, which is a set of tools to help users migrate. And this Includes. You could be coming from elastic 6.8 or elastic 7.1 or something like that, or OpenSearch 1.3, for example. These,much older versions, the um, there'sa number of different strategies there. Uh, but ultimately you cannot just simply update the software in place for those older versions. You're going to have to move data and potentially reindex data. Um, but themigration Assistant has,a set of tools to,make things easier for you. It also has tools for um,uh, helping you sort of test out compatibility ahead of time. Um, possibly often a strategy, particularly for the cases where high availability is a big deal, you're actually going to want to stand up the new version, mirror your data into the new version, have your old version running, and then kind of test your application against both the Migration Assistant will have various toolsfor potentially even capturing some of your queries, making sure your queries maintain compatibility, all sorts of that. So I would um,anybody insort of that boat, please do check out the migration assistant, uh, for tools there. And then also, um, AWS has lots of documentation about, um, updates and strategies to use for,upgrading as well. So yeah, lots of resources available there. So I'm going to you already answered this in that uh, in that response there. But I am going to call this out separately once again. So for anyone who feels like they want to jump into OpenSearch now, they're currently on elastic, do we have a path for them to get up to 3.0? We do. Yeah. Um, so check out the migration Assistant. Um, but yeah, wedo get asked sometimes like, are you compatible? Or like I have a client that's talking to elastic. Do you speak the exact same API as Elasticsearch? The answer is no. Like these are different products, right? Theyliterally forked. And at the time of the fork, they're sort of independently evolving. Since then, however, we do have, um,you know, lots of these APIs are fairly slow evolving. You know, things that work don't need to change. Um, and we have tooling to,help with these differences and enable the migration. Yeah. And to we talked about this on the last episode, but for people that maybe didn't tune in to that, we did talk about this having some breaking changes around plugins. So can we talk um, briefly about, you know, migration is one thing, but then also you've got the add ons, plugins, um, and making sure that those work. What does the strategy look like now for plugins going into 3.0? Yeah. So I mean theplugins that we ship with our distribution, um, those, you know, those will,all work and you know, the,features, they'll still continue to offer all,the great features they provide. However, we do have lots of users that develop their own custom plugins that they install into OpenSearch. And so it will, um, you will need to update your, uh, so your custom plugin will be its own sort of development environment. You will need to take a dependency on the3.0 version of OpenSearch, and you can expect to find some compilation errors and things, and you will need to make some changes. Um, many of the, uh, we talked a bit about some of the modularization work. So many of the problems you'll run into are kind of just structural changes of code has moved around, so you'll just have to change some import statements and it will still work the same way. There are depending on how sort of deep into the internals your plugin gets, you may find more significant changes. Anddue to some of the platform updates of requiring a more modern version of Java, the changes with the new version of Lucene that come in those may require somemorechanges. But again, those are the very advanced users that are actually developing their own custom code to run inside their OpenSearch clusters. Not everyone's going to see this. You know, side note here. If you're not doing your own custom development work, the default plugins that ship with OpenSearch are going to work. It's if you're doing additional. Exactly. And even if you're doing some additional code change or maybe developing your own plugin, I think the whole release cycle of a major release is tuned into like alpha, beta, and then GA as compared to any minor release. So I think this is again building in the perspective that, okay, we will ship on the Alpha and beta, which will be an opportunity for any partner or plugin developers to kind of try out these, you know, changes that they need to do. If the plugin needs some of those changes by the time GA is launched and get that ready in time. Yeah, the artifacts, the these alpha and beta artifacts that are available in Maven to pull down. And so early adopters that want to get started on this work early um, it's been available for them now for a while and will still be there. So it so there's no need to wait until the actual GA release to get started onthis early integration. You can start right away. Saurav um, as you know, people are testing this out. When something goes GA, that means it's obviously we've had other alpha releases, potentially beta releases. Um, have you heard of any success stories? You know, this is the kind of thing that, like, engineers like to hear about. They like to hear about how other people do things and the successes that they've had. How that's done. Are there any stories orum, things that you've heard of with these? Um, you know, prior to GA, uh, successfully migrating over to 3.0? Right. So I think, uh, there are a couple of stories that I can mention here based on the feedback. I think we got some anecdotal data points saying that, okay, a 50 node cluster was successfully migrated from 2.19 to 2.0. It was when one of our issues as well and forum. So that's kind of, you know, tells that okay. It's not uh, something alarming or nothing, something uh, a lot of consideration is not needed for users when they are simply moving their workload from 2.19 to 3.0. Other stories, I think, are related to some of the features like read write separation. I think we talked about that previously. It's one of the sort of feature where people are looking for isolation of workload and, um, failure and things like that, with having separate deployments for readers and writers. Uh, so we have some of the partners who have already been trying this feature in the experimental version, and they have submitted us provided on GitHub with good feedback, which we iterated upon. And then with 3.0 will see the GA version of separation happening. Awesome. All right. So last question. Then we'll turn you loose here. What has been the most exciting feature or functionality in this whole process? It's been coming for a long time. What are the what's the thing that, you know, the feature functionality that you're most excited about? We'll start with you. Sure. I'll take this asa developer and engineer. To me, the most exciting thing is, um, the update to a more modern version of Java, like the Java language, has been evolving and lots,of great new things havebeen added to the JDK. And for the OpenSearch 2.9, we needed to maintain compatibility with Java 11. And so that meant a lot of these,great new features that make development a lot more fun have just not been available to use inside OpenSearch. And so with 3.0, we're moving up to a more modern baseline of JDK 21. And so that unlocks lots of great new syntactic sugar. And also just like new performance and features within the JDK that are really exciting to use. And so, um,even early on, we're not necessarily leveraging everything, but like it's this is more of a forward looking thing. I'm really,excited about what this will unlock for the project going forward. Yeah, I think it definitely it's a hard pick between some of the foundational changes, what we have been doing, uh, and with realizing them in 3.0. Uh, but I would pick up like maybe Lucene ten. Integration and upgrade is one of the most exciting things that I have been. I'm looking forward to. And it's primarily because coming from the search engine area for years, I feel like this is like getting a new shiny engine, like under the hood, and then you get a lot of these improvements out of the box. And because I have been also closely working with some of the folks here on the team, on looking into some of these benchmarking numbers, going diving deep and understanding what Lucene is offering. Lucene ten is offering not just for the search, but also for vectors and other related use cases. So I think it's going to be a huge lift. Awesome. Well, once again, thank you both for joining me for this interview. I appreciate your time. Looking forward to OpenSearch 3.0. And for everyone that's still here watching, please like and subscribe. We'll see you on the next episode of INSIGHT links for now. This is Brian Graf. We'll catch you later. Bye