Scaling OpenStack Development with git, Gerrit and Jenkins

so I like I said we’re both work on the OpenStack project development infrastructure the short version of that is that were the people that buy our from our point of view make everything easy and automated for people and probably from the developer’s point of view get in their way from doing all of the leap fun codes coating things that they they really want to do I’m sure they all of us we’re going to chat a little bit about some of the great automating type of things that we do to have the two of us be able to manage a community of hundreds of people so to start off just a little bit about OpenStack 12 exact is actually a i would say loose federation of fraud projects and there’s a couple of the folks in the project who would like for it to be looser but we have we have multiple distinct projects at the moment there’s six core projects glance which is the image service which somebody as a quick step back OpenStack is open source cloud infrastructure software that is that is used to create your own cloud I realized that I might not have mentioned that and it’s possible somebody hasn’t heard of OpenStack it’s made up of several components it originally started off with to Nova and Swift Nova being the the compute service so not to use other people’s marketing terms but if you’ve ever heard of amazon’s ec2 product it’s the thing that does the thing like that spends up servers for you and Swift which is the object storage which is sort of similar to Amazon’s s3 except different because it’s not same thing but is also where you can put data in addition to that we’ve now got the image services where you would upload your your machine images horizon which is the the web GUI tool kit excuse me dashboard quantum which is networking as a service in Keystone which is authentication as a service there’s several other projects that are in incubation at the moment and and the list will continue to grow over time which means that when we’re talking about doing development infrastructure we’re not doing element infrastructure just for one source code repository or project but similar to some other people out there Android being a good example we’re actually a collection of related repositories which makes things all the much more fun so a point about our contributors are lovely clients as it were we obviously coders are themselves individuals turns out that companies don’t write code as a unit they are made up of individuals who who write the code in the case of OpenStack I don’t know how specific this like I don’t know the thing one of things we noticed is essentially we have very few people hacking an OpenStack who are doing it in their spare time at home it’s almost all corporate contributors I’m sure there’s a exception to that rule that proves the rule somewhere but it’s it’s certainly we’ve got a lot of people being paid to work on it and for that reason there’s there’s two important things to that we have to take note of one is that the the number and quality of the contributions that we get varies wildly we have all of a sudden I’m also the number that we have in the community could change at any moment AT&T recently just announced that they were going to be a part of OpenStack and for all I know they could show up with 150 developers tomorrow you heard about that when we did yeah exactly it’s like a couple as a webinar I think that somebody did it somewhere and all of a sudden apparently 18t is playing with us now I have no idea if there will be 10 or a thousand developers from them but I sort of got to be able to handle that it’s a sort of unexpected scaling a situation from process perspective a little bit about how we how we orient things we started off with a lot of us having had some of them to experience so we took on the time-based release idea from from a boon to we release similar to them on a six-month cadence and we actually release a couple of weeks before eben to release to assure that the latest OpenStack is always in the latest ubuntu we and we’ve been we’ve been working with them pretty closely from from the get-go not to leave anybody else out we’re always we do work with people from the other from the other distros but at the beginning but later we’ve designed some it’s where we all get together again this is taken from Canonical’s structure we get together at the beginning of each development cycle to plan the release so that’s two meetings a year again a couple weeks right after the release or before the beginning of the of the next cycle our trunk is continuously open we don’t ever close down drunk and we do through Garrett and through the stuff we’re going to talk

about development is done directly on on master we do not have long-running feature branches that are public if you do things on your laptop whatever I don’t care but you’re submitting your patches for review to go to land directly on master and then we do we do testing to make that work we we do one month releases every milestones that are looking forward towards the next release and then once that once a once we have made a release then we make a stable branch of that that’s there for ongoing maintenance and those are actually typically turned over to a group of people that are composed comprised mainly of the folks in the distros because once we’ve released it at you know we won’t look here but actually that’s sort of a bit of a success story in that wasn’t in the original design for the project but people from several distributions came and said we need a place to collaborate on these patches we don’t want fedora to apply this patch and and and Ubu not be able to take advantage of it so that’s sort of something that that the district aim up with by themselves and we helped facilitate that as a project yeah so as far as that goes we actually the core the core teams for Nova being the sort of where a lot of development happens the core Nova team who have review rights on innova actually don’t have review rights on the stable branch of Nova the the folks who are doing the distribution packaging work actually essentially own it at that point we’ve released it to the release it to the world and now the world gets it so so that’s yeah that’s a really fun thing so the the underlying vision behind the tooling and process that we’re talking about here has a couple of elements and it starts with it starts with the idea of consistent tooling we’ve got it’s constantly in in a group of hundreds of developers like herding cats everybody has their own their own ideas they’ve got their own opinions on how things should be done and everybody’s got their pet tool in the in the corner I don’t care because that’s not that’s not really scalable because then we get into VI versus Emacs arguments so as much as possible we keep a consistent set of tooling across all of the projects so that people can easily contribute to any of them and they know how that works which leads into the consistent process the process for contributing across any of the OpenStack BOTS projects is the same so you come in your work on one of them you can work on any of them and you know what to expect you know how they’re going to be laid out you know how everything is going to work and the the intent is that that leads to a consistent product we’re we’re not testing one of them differently than we’re testing another one we’re not running one of them through a different process than another one there’s not some guy off in the corner who each does his own thing we sort of know what we’re doing and because of that then we can we can reap a multiplier effect from not just having 20 developers working on Nova and five developers are working on Swift but we actually have 25 developers who fully have the capabilities to work on the project as a whole which which I believe gains us a lot more traction moving forward with the you have a consistent tooling like I’m one of the things that we’re trying to get rid of there is a lot of people doing a lot of meta development what i don’t need is somebody in each project working on writing their own set up by files it’s it’s actually just not interesting work and and typically somebody will go off into the weeds and like come up with all sorts of crazy crazy things that might need to be done to create a tarball and and that’s that doesn’t help anybody and then nobody knows how to do that once they get hit by a truck or or just you know take another job as we get processed process diverging we get developers wasting their time either either working on little little pet tools or on trying to figure out how to work with another project and that’s that’s also just a waste of energy we’re with lowers keeping Atkinson set of tooling lowers onboarding time and especially when I seconds tooling also should be really clear I to the to the best of our ability we try and keep that to be reasonably standard I don’t want to build our own build system that’s I’ve worked on those projects before right it is so much fun everybody wants to do it just like everybody wants to write their own rest library and their own web framework and configuration management’s their own configuration management system we try and keep it as 10 as bog-standard is humanly possible so I want you to be able to grab a OpenStack source repository and do standard commands with Python setup I without having to read a readme file to learn how to interact with our source code repository and have things work pretty much like you would expect them to work we’ve got a couple deficiencies in area we’re trying to fix but anyway so project specific weird build could we try and get rid of as much as possible this is oh wow the even scrolled off the screen we have a lot of these this is this is sort of a when I talk about development infrastructure systems this is sort of what I’m talking about the main two are Garrett and Garrett and Jenkins Garrett is where we’re doing our code review on and we’re going to talk about that a lot more in

detail in a second that feeds into Jenkins which we’re using for doing testing we do testing before merges in dart into our chunk we also do some testing post-merge for things that aren’t quite stable enough from a testing perspective to be able to do pre-emerge we also use that to do post-merge artifact management things like tarball uploads translations processing things of that nature for bare metal deployment testing we use orchestra from a boon to which is based on cobbler for Medhat one of the weird situations where you’ve actually got a tool that was developed essentially by two distros and if like if both red hat and and a boon to hurt you are working on the same tool it’s got to be good right but anyway it’s been really it’s been really helpful to do that and we might get of that here depending on time we got we as launch pad for everything that isn’t code review in terms of managing the project so bugs blueprints we released our balls to that we manage translations there it’s the open ID source of our single sign-on system group permissions in terms of who gets to who is able to review things and Garrett it’s all managed through there we also manage documentation servers IRC BOTS we’ve got a planet a planet based blog aggregator package repositories run ether pads all of these things or tools that we try and provide for the for the community so that everybody’s got what they need to be able to do but also so that we know what it is so that if something goes away we know how to fix it because the last thing you want is somebody to start depending on something then it it’s not there anymore a couple of these things are actually on the to-do list because at least two of them are running on some random person’s machine and I have been asking for a couple months where where they’re running from so we’re trying to trying to keep that to a minimum as well our consistent environment at the moment we target all of our development platforms target of into just the latest ubuntu release is is the is the basis which doesn’t mean that we don’t care about the other the other distros or the other releases but so that every person’s coding doesn’t have to take into account up into four releases have been to Fort releases of fedora arch gen to mend Riva whatever we try and keep that as a basis and then luckily the Mets Python there’s not that much divergence everything is in Python we made a decision in the policy board a while back that we would we would maintain a consistent programming language unless somebody comes up with a reason why particular piece can’t be in that program in language so far nobody has come up with a reason why a particular thing can’t be written in Python I know of a couple who are about to try but we’ll see how successful they are at that for the developers we have a local testing system that they use a virtual land based virtual engine based tests we have all of our development writings done and IRC there’s a tool dev stack that some guys wrote that does basically local deploys into a vm on your on your machine so you can do local integration testing which is also the basis of the testing that we do inside of jenkins pre pre merge that’s actually it’s a it’s a really good way to sort of start to play around with OpenStack as well it’ll-it’ll set up an OpenStack installation on a vm for you and and you can play around with it and it does it with get checkouts of all of the components so you can you know control see them hack on them a bit and restart them right inside of a screen session yeah and it’s an annotated shell script which at first I was like oh it’s a big annotated shell script but then that’s actually sort of easier to read than say a large repository worth of chefs chef recipes for a basic install so there’s a reason it’s called dev stack yeah not intended just to be really clear not intended for production deployments please but the love of all that is holy it’s it’s intended for you to use on your laptop to do so that so that you don’t have to learn everything there is to know about releasing a large multi-node system to be able to do tests local development testing anyway and then the thing we’re gonna talk about a lot is the thing I mentioned earlier we we run a gated trunk which means that which means that that developer a developer’s process which might be the next thing oh no yeah we do a we do all of all of the commits that are going to come in get tested before they land on master and if they don’t pass the tests they don’t land so there is there’s never ever commit ever in the history of the project that has landed on trunk that doesn’t pass at least at the moment unit tests we’ve been working on adding integration tests to that but but literally commit one we set up the gated drunk before before the commits started rolling we’ll talk about the mechanisms of that the reason we do that is is as many primarily its to ensure code quality right I don’t care how much you really want that feature and if it doesn’t work it’s not landing it’s that

simple but one of the other things that it does is it protects developers right because if you’re gonna do development you pull trunk and you start doing your development and then you run the tests and they break guess what broke it your code guess who knows how to fix the thing you just wrote you so as opposed to you trying to do tests and then it’s broken and now olson you’re trying to debug somebody else’s code which you don’t really even know their intent how can you possibly be expected to do that so that’s one of the really big keys that we get out of this is that it protects the pics of tree you don’t have to worry about the the tree being broken so we don’t have that wait till Jenkins is green and then you can do something it’s always green if it’s not well it’s not green there’s a big problem and another thing this this comes from some past experience two things it’s gala Therrien we don’t have anybody to special you don’t have a guy with ability to push code directly into the repo bypassing code review its it all goes through a code review system it all it’s all peer-reviewed so you don’t have like the random maverick developer who decides he’s going to do whatever he feels like doing it’s it’s the same process for a person working at Rackspace as it is for a person working HP as it is for a 14 year old in his bedroom that feels like he wants to get into cloud computing don’t care he submits code the patch gets reviewed it gets in that’s it so that’s the that’s the thing those are the things that we get out of that and one of the ways that we do that from our perspective since there for a long time there was me and then recently over the last six months there’s also been Jim so that I don’t go absolutely crazy it’s all automated because when you’ve got enough developers working on things if it’s not automated it’s it’s just a failure of most of the things that a human might want any any human interaction and this is just an opportunity for failure and almost never has an opportunity to make things better but you can’t like land a patch better like yeah I committed the crap out of that I’m pretty pretty sure you can land a badly though yeah you can definitely hand it badly so if it’s if it’s a thing where the only human option is to mess it up it should be automated and so this is a picture of our nice green Jenkins that that runs might be a little bit old but uh you know she ignore that so the way that the process works to get in that is that code a delaford checks out code from master writes it tests it locally in a virtual end of using tools that we’ve got on the tree they submit it to Garrett for code review its peer reviewed it’s either accepted or rejected at that point it is run through automated checks in Jenkins as soon as the as soon as the the code of acceptance lands if those checks pass the code is merged if those checks fail the code is rejected and sent back to the developer 444 re-review yes glad you asked because that’s a huge security hole because basically you’re just letting anybody submit code to a system and then run on your build farm without anybody having seen what it is we we have the the cooperation of a couple of large cloud service providers and our Jenkins farm runs quite a few Jenkins slaves and for some of the tests we’ve built up build machines on demand and run them so you know it’s it would be a useful target to take control of our of our Jenkins system and use it for a nefarious purposes yeah also the the other reason past security because there’s there’s optimizations that we can make to the process so that if we know who you are say you’re a core committer I might be a little bit more assured that you’re maybe not doing a botnet in a unit test right I might know who you are you might still be doing that but like I might be a little bit more so you could have a bad day you could have a really bad day and be really upset however there’s another thing which is that I I don’t care if you wrote good code right what i care is when i apply your code to the tree is that going to be good so if I test it when you submit it that I’m testing what you wrote and that’s cool and that makes you feel good about the quality of what you wrote however other commits might have landed in fact they probably have landed since you submitted and so what I need to do is I need to test right before we merge because otherwise I’m not testing I’m not testing the potential state of the tree that I’m about to produce and so there’s a race condition in there that most of the time it’s actually probably gonna be fine but this is me dancing about about how the most of the time doesn’t get you the times it doesn’t so anyways it that’s I think we’ve got a slide about this but the normal the more normal Garrett approach to do this is actually to run tests on submission you know I think a lot of that comes with a lot of Garrett installations are likely private so you know if if you if

it’s completely closed you know that all your contributors are working for the same company then obviously that’s not going to be quite the problem but we try to be as open as possible yeah any anybody literally well okay you you have to sign a CLA because evil but you have to sign a CLA so there is there’s one step you have to go through before you can submit code but I still don’t know anybody can sign a CLA you know it’s like that’s not hard so anyway so that’s that’s the general process flow we’ve been talking about a little bit you want to do though yep so Garrett was originally developed at Google for the Android project has actually developed at Google several times yeah it started off as rietveld if anybody’s ever used rietveld and then it and then it migrated to being the Python version of Jenkins so Garrett if you check out the git repo and you reset yourself to tag 10 you’ll see a Python implementation of Garrett and then they rewrote it in Java which how many times that’s happened in the history of mankind I don’t know but so it’s it’s a standalone patch review system it doesn’t have a bug tracker or anything else associated with it so actually a lot of what we do is integrate Garrett with with other tools it’s it’s kind of nice in that it it has a lot of flexibility and a lot of a lot of integration points for other other tools so you can it calls hooks on events you can do Jason queries against it and you can ssh to it to its pseudo ssh server to nada you know it’s not an actual shell and it’ll spew an event stream at you and you can watch everything that goes on in real time so we actually in addition to the the things that we’re running other people in the OpenStack community have their own tools that they’re building on top of Garrett to facilitate their own unique workflows so you know those aren’t required for contributing to OpenStack but it’s it’s great that we have a tool that’s open it up for people to to build things on top of like that in fact in in a continuing stream of answers to the question from earlier what we we have a guy who’s doing on on submission tests of and he does he does integration deployment tests on a couple of different systems for every every patch that’s submitted before people review it he is he solved the security problem for that by just not caring but that’s fine because it’s his like that’s his prob knew that it doesn’t it isn’t really and then that system actually just reports comments back into the garrett system like any other community at anybody can leave comments on an honor of you you just come in sign in comment away doesn’t have special privileges but that’s that’s great it’s great that these written that and that people can use that you know and garrett has extensible review categories by default it has the idea that a change will be verified so somebody in fact i think probably the most simple garrett installation is that somebody tests each change and says yeah just that and it works so then the next thing is you hook it up to jenkins and it does premar checks we hooked it up to jenkins and sorry do patchset uploaded chats we hooked it up to Jenkins and reemerge checks and then there’s the code review category as well we’ve actually added a category 2 2 hours called approved so there’s sort of a two-step review process where you you you know change collects reviews and then somebody decides it’s it’s time to go in and they they they set that the interesting another interesting thing about our installation is that jenkins is the one that decides that it gets merged so that’s actually something that’s a little unusual for for garrett did you know all of the people come along and decide whether it’s okay to go in and then Jenkins has the final say as to whether or when it goes in and that’s that’s actually a serialized process so as soon as that starts we know that the tree isn’t going to change via any other influences so we can be very confident of what Monty was saying earlier about how you know that the change that the tree was tested exactly as it is when that change goes in the trees change while he’s going it just starts again Oh optimistically issue the patch is gonna be erotic just take everyone go Adam then they stood a chance well so the the merge jobs are not permitted to run in parallel so they get queued up right now Nova’s Nova’s test suite takes about 10 minutes to run and we actually we lucked out that it takes about 10 minutes to install devstack as well so that does those run in parallel so we actually have you know like I think there’s four

merge jobs that run for Nova in parallel but but the changes themselves are serialized because we don’t let any one job do that and you know right now waiting for 10 minutes for your change to go that’s that’s not a big deal for four people yeah yeah we may have to revisit that when when that when our testing gets that good we’re also we’re also hoping I think when we get to that point because because we do have donated cloud accounts from two major cloud providers means that the spinning up machines is a thing I don’t even think about I can look like water so a one of the ways that we’ve discussed as test as the test we starts getting blown out is ways to slice that so that i can run part of the test suite on one machine and part so i like it if the unit tests start taking a half an hour i could maybe spend up three machines and run a third of the of the tests on each of them so try and parallel why’s that so for each test i might spin up my twin again from spinning up 50 machines but who cares that’s not my machines so every everything that we do all of our configuration of our systems is is either documented or or in like puppet modules that are in get on github that you can check out so ostensibly anybody can can do exactly what we’re doing even to the point of copying our config files now granted possibly not everybody can spin up an infinite number of cloud servers without getting our algorithm I promise we didn’t design that to drive business to our employers its course not never would have thought of it out I mean if you could do that why wouldn’t you right right oh yeah so we did in fact we’re working on a i’m just going to diverge here we’re working on a there’s a there’s already a Jenkins ec2 plugin that a lot of people use to do on demand things and we’ve got a guy working on a similar plugin based on the java j clouds library which is a pan provider so that you can use so if in my case where our provider is not ec2 in his in fact Rackspace that I can have Jenkins spin up those notes for me so that’s that’s the thing that we’re trying to push forward and get that and that obviously will all be might not be obvious I should say that also everything that we do is is open source and any patches that we do to any of this stuff the only reason that they’re not upstream at the moment is because we’re still having to go through some paperwork to get them landed but the intent is that there’s nothing that we do that is private to us I Garrett Garrett Garrett just came online last week so yeah the colonel adored issue made it I mean yeah understandably obviously it’s tricky so here’s actually a transcript of a review of as a UI thing Garrett sort of squashes review text if you’re you know if if if it’s boring but so the idea is that people can leave comments on this and eventually somebody decided that this should go in and Jenkins started doing these merge checks and they all passed and so they got merged those are all links directly to the Jenkins jobs so if something fails it’s easy for a developer to right to it look at the console output and realized that they didn’t put enough logging in yeah that’s never not happened before so here’s here’s an overview of the review page where if you’re running a standard Garrett you’d only actually have the V and the are columns on the right that’s for verifying code review a is our approved you know it’s ready to go in thing it’s an exercise to the viewers to figure out what’s going on with that change in the middle that’s verified approved and rejected we we know the answer but if you want to if you want to find grab the idea of that and go dig around in our Garrett and figure out what that what happened there you’re more than welcome to you I think to mention about why we out of their approved column is that in the standard Garrett a we have we have a policy that two core reviewers should review a patch before it goes in and there’s not really a great way to visually see that that has happened for another court reviewer because the the imac or reviewer and i’m going to give this the the extra plus to vote in the code review thing just indicates oh I’ve and before that was actually triggering the the the the Jenkins so we did this so that so that the core viewers could vote as a core reviewer and we can see two of them and then somebody could choose okay I’ve we’ve reached the threshold this is going to go in so apparently we have 10 minutes and 14 is significantly less than 26 oh my god faster yeah oh so there’s a there’s a plugin for Jenkins called the garrett trigger plugin and it’s wonderful it

lets it ssh is into Garrett and watches that event stream that I mentioned earlier and so the version that we’re using actually lets you trigger jinkins jobs on somebody uploading a patch set a patch set getting merged or and this is a change that we made locally and we’re about the upstream somebody changing the review state of that patch so yeah Lou said earlier we did we use open ID from launch pad since all of the group information is there and lunch fat supports an open ID extension which exports group information is part of the open idea interaction so jinkin excuse me Garrett already supported open ID but not in a it supported the hey tell me what your open ID is man sort of way and so we put in the support for us saying no in fact that’s cool but you need to log in with your with your launch pad open ID and we have 0 because of this we have single sign-on for the project for Garrett Jenkins our wiki and I thought whatever else we want to throw at it yeah so we did bug integration with launchpad so we we do troll the wheat roll the commit text for for bug reg exes we make links into the bug in launchpad in the in the description there and then also will auto generate some topics here but then in launchpad will also come in anger or register with lunch bad that there is a review up automatically and it will change the state when a winner when a developer uploads a patch to gear for review the references a bug it changes the state to fix committed and it’s going to be work in progress and then when the when that patch lands it changes the state to fix committed so we get those workflow transition so people can track it but the developer actually doesn’t have to spend a lot of time going to bug trackers and changing the states of their bugs because that’s sort of wasted time for developer honestly but so we can we can track that there and we make sure that we give links to both the commits we do a read-only mirror of the code to github which makes it easier for people just to branch it we don’t do anything else with github but it’s a good place to publish and then we do and then links to the review if you want to track what what’s going on with that do the same thing with launchpad blueprints which are another way of planning virtually the exact same idea we look for their books that have yeah what’s that yeah they’re bugs that haven’t happened yet blueprints are things you’re planning for a bunch of things that happened unexpectedly and same thing we link into as much information as we can into the end of the blueprint on launch pad so that you can track what’s going on with that Garrett supports an idea of topics so you can you can upload things and tag them in a particular in a particular way we’ve got some tooling in our get review tool which is talk about a second this is a topic with one thing but if you if you had multiple related changes that were two to a sort of a longer term development thing you could track those inside of carrot in terms of seeing progress and speaking of good review so the way that you submit patches to Garrett is you you push them there via get which is really cool you push it to your ref specs but you it’s just to get push command that does involve a bit of a learning curve for people to learn how to use the various Garrett features by pushing to various weird URLs inside of get it’s very cool you can totally do that there’s nothing that we’ve done that that breaks somebody’s ability to knows how Garrett works to do all of the things you can do with Garrett however the fine folks at gluster had a tool they also use Garrett had a tool in their source tree called RFC SH that made it really it was a run this script and it will do the it will do the appropriate incantation with get to submit your patch up to Garrett we took that and put it into our trees at first and that was great we’re like yay we’ve got this thing and then I don’t know if you noticed before but I mentioned we’ve got six core projects we’ve also got some branches of them stable branches of those we’ve also got client libraries we’ve also got ancillary projects I got really tired of copying this script into every repository yeah and then if we made a change to it heaven forbid now we’ve got to update 15-20 repositories we never did wait what we never updated it yeah I’m just wrote get review instead that’s true yeah so we wrote this so this is a this is it simple on you can pip install it so pip install kit review we’re working on getting into both fedora and Ubuntu but so it actually presents as a normal get command get subcommand you just type amazing enough get review and it does all the right things if everything is magical actually we stuck a dot get

review file in our repositories with all the configuration information about where to find Garrett so if you’ve never submitted to Garrett before the first thing it does is it sets up your repository correctly and then got submit to your change to the right place and typically it also it will also unless you unless you tell it not to it will rebase your change for you on top of on top of the tip of the thing that you’re submitting to so that your patch is actually the thing that you think that you’re submitting and this is this is why it’s a good idea to use a program like this because if you didn’t then you would be you know get push the address of the garret server refs for master but whatever yeah so it in addition to processing the bug of the commit text for bug and blueprint information we use that to make Garrett topics for people if you don’t want to put a dot get review file in your repo because there’s some people that don’t like having that sort of thing in their code repos the only thing you have to do by the only by hand set up step you would have to do is to create a git remote called Garrett with the location of your Garret server then all of the get review stuff will work magically like that so this has been really helpful and we’re able to add you know various features we added a feature to this there’s a dash D flag to it that you give a garret change review ID and it will download that change into into a local branch for you so that if you want to look at the code locally you don’t have to you know know how to do all the various things you just I want this in AU grabber for you and then you can do what you want to with that that’s really cool we push that back to the sense of messages to the gluster folks to let them know about them we’ve got a couple other projects that have started picking that up so yeah so there’s nothing open stacks with specific in this if you use Garrett you can use get review yeah and actually if you want to make it support another review system that’s cool with us too yeah sure take a match for that yeah our get review is managed in the OpenStack Garrett so you can get review changes to get review of course so how are you doing on time couple three and a half minutes we can walk through so that’s uh wait so we’ve got we can walk through what we is that bridge about there yeah so we test a lot of things that we we on on all the commits right now we’ve been running unit tests from the tree we’ve been working on adding functional sort of in a functional integration tests with the idea that there’s some things that we can spin up virtual servers for do it do a single server install of that and do a sort of post install does this work testing and then there’s other things that we have to do bare metal deployments of and so we’ve got some machines they’re not currently operational to moment the process works there’s a there’s some other issues in there but that we would do a complete wipe and reinstall from scratch on to undo bare metal machines and then deploy that revision of of OpenStack onto those bare metal machines and do that testing and there’s some there’s some issues in there with both of these one of the things like the functional tests are easy for debster run locally you can use devstack on your laptop and run the tests and do it and so that’s we actually run that because what we don’t want to run a lot of stuff on our Jenkins infrastructure the developer couldn’t reproduce locally because then it’s really tough if it fails them for them to reproduce the bug and fix it so however one of the things that we have been developing and it’s not rolled live because we haven’t gotten permission to roll it live yet one of the problems people giving you free resources is sometimes you have to ask them permission to do things with them is Jim wrote some excellent code that on failure will basically install the developers SSH key on the on the vm and hand it to them so if their test breaks the if their code breaks the test here here’s the vm with the with the your broken system fix it or use this to debug the problem and we’re going to delete the system in 24 hours so no they’re writing bad code to get these servers yeah that’s if you can automate that so and then and then we probably won’t be able to do that with bare metal unless we had a lot more bare metal anybody have a lot of bare metal you have somebody wants to give us a lab of like a thousand bare metal machines then we might be able to do bare metal deployments and had you log into that but I’m guessing that’s what I’m not going to happen any time in your future what’s up hey you got some unity companies that make machines so awesome great anyway this is weird the other thing and if we were here for another couple hours we talked about this but where we’ve been we’ve been trying to structure the way that we do our testing in such a way that um the very there’s open zach is a lot of options of how you can deploy it and a lot of people have vested interest so you’ve got you know HP that has their interest to rackspace that has the things that they’re more interested in or cisco and citrix i can’t possibly test I can’t possibly be responsible for all of the combinations of those but if you’re a vendor who’s working on something and you want to hand me a lab

you know actually not have to hand me the lab if you want to have a lab that’s configured to test the configuration you care about and hand me an entry point to it I will happily run whatever like I’ll trigger runs of tests in your lab and is your lab you’re the one who cares about the thing so you can define what the test is because I don’t care but if you’re really interested to make sure that you’re crazy Cisco switch works then you run whatever that test is and if it passes then then we’ll report that into the central Jenkins and if that’s really solid we can start gating trunk on that too exactly so that you make sure that that random developer from you know your competitor doesn’t accidentally you know tank your thank your code so we’re trying to we’re trying to make that azz vendor-neutral and be a centralized point where all of the vendors who are participating in this can can sort of collect and coalesce there’s a integration test project called tempest yet another thing that we have that we manage in getting that’s a whole other thing we could babble on about that for a while but I think we’re probably out yeah we’re out of time so great that’s what we got because that’s time thank you everybody