Henning Brauer: the surprising complexity of checksums in TCP/IP (P7B, AsiaBSDCon 2013)

here I except this only there in ipv4 ipv6 does not have that that is a lot of advantage but I’m getting for later um be Jackson’s checksum algorithm is actually very very very simple all that’s done is built the sum of the confidence of all words to the parents or the section 12 checksum and take the last word of that super easy the IP Jackson only covers the IP header we will see what this is important does not have a table while the GC communicate Jackson does cover almost the entire effort specifically the sheets the payload it does not include all of you are kierra includes of Qi Pierre we see you’ll see that it comes from the vice fine idea layer it puts the anti for another and send ball of payload the false consideration we’re talking about check something I mostly care about the forewarning case then we talk about about the case where the connection is no can be terminated or central Real ID or something else the total cost of the packet processing in the colonel doesn’t matter anymore because the other stuff is much more expensive in comparison but Hector processing of the phone is prepared to start the google and pretty cheap the map to do the checksum is dirt cheap can consider free I spent almost 10 years now and profile integral of finding Eric spending the time and why buy the web for example from so I were pretty good idea of what’s going on the confirmation that said when you consider free community with tighter the prophetic often the cost is a different one cost with latency and Pam go to the memory in modern CPUs you are completely there must be popping ABC’s for /i 36 here it’s also true for other modern designs you are completely incapable of getting the action integer units to be busy it’s basically impossible effects never every illumine the limit always is some kind of latency sometimes bandwidth usually latency can be the latency between lead the decoders that take your sources instructions and decode them between the little micro ops they work with most of the time it’s the latest into the caches but it’s tough is a lot of cash and you have to talk to a memory then the CPU is idling for several thousands cycles memory is really really small compared to the speed of the significance of smell and also um the other another case to consider it is for latency to the device you’re talking to do so if you’re talking to to the network they’re talking about ladies on the pci or pci boss is a waiver in open hearts on rice so that’s organization check something hot data stop that is in hatch is basically free you do not even find these things in the profiling brass always somewhere very down below it doesn’t matter it’s free since the IP chisun only publish the header and to my definition have the header in hand put forward the packet be cursing the other information right that is free which is why not having to check some there is not in 446 doesn’t make a difference that’s the model advantage the we have to check some data that we have to fetch for memories that is expensive to first assess the CP google i/o for a long time waiting for the day of the show the TCP and UDP checksums covering the entire payment have exactly that problem you have either a boolean fill the gaps no problem the TCP header or the UDP header lightly the payload is almost guaranteed to not to be in cash as had the local determinated connections are completely different matter but those of a matter of the first end of the cost of the local processing Apuleius naked for most

people that’s so high that this is there some platforms have optimized strikes on code this is actually kind of interesting I 362 64 and SH have assembler implementations for the Jackson on room and a couple of other platforms have optimized see versions that might contain a center focus wallet in check on one um oops it’s a very good question whether the optimized phones are actually hostile you have many many cases where the cemetery publications were slower than implementations because they were 20 years old all their assumptions about cpu architecture internal workings of the make assumptions that are just not truly more and the compilers we can muster so it would be a very very very good thing to actually measure whether these optimized versions are still faster than the generic see version they have that’s a project for somebody of you I want to do this easy so please do it so speaking about the IP checks on that’s the anti IP header as you can see if it’s it’s pretty small right there’s two things in here that we care about for the checks on now the checks on itself sits here and you Jackson has to be updated but each and every router because the time-to-live is updated for Internet routers which I suppose to recalculate right and set since it only covers the header the header refresh anyway motor performance canonical forms problem at all pretty much each of every modern network interface card have the offload engine for trying something for the IP check so that basically does not make a difference of course well as I keep mentioning highlighting it is basically free the checks elimination of the ministry and the flow will caramelize three times so we’re taking the shortcut here basically for the layers I hear about they’re talking about I PA sport following IP forward to an IPO in IP input we have the decision whether a packet is globally delivered or to be forwarded on or before ready for tomorrow and right after that there’s the PF tesco that’s the entry point into TF so PF is there if it’s local its hand up in the upper layers and I said I don’t really care about that if it’s in forward it goes right before word in I before word we to the routing kind of the rocky look up actually happens like people but that’s a different story people TTL recommendation and in case it’s on route will generate the acpi off we sent back to the source in IP output is a longer or none of these most of that code needs with locally generated packets that do not have the head of heels filled in at that point so I the output that’s that surprisingly it also calls PR test for Alabama PF chang and right after that call to Kiev test the IP checks on this recalculated unconditionally always you have to recalculate the papers of the TDR the presentation right locally generated passes at that point on even have a check some pointers after PF which I some is retail forces we have had one code to deal with the architects on whatever we did code and I we rewrote something in the packet like mad or redirects or the scrubbing stuff the update of the Jackson to compensate for the strangers right well that was completely pointless because the tech service replaced afterwards so that Coach but almost almost almost just remove their code why not well there’s a case to consider the bridge but you’re bridging you’re obviously not really looking at the IP header all we actually kind of do if you want to figure bridge is supposed to do layer to it isn’t really fun it’s supposed to easily or two and the bridge does did did not recalculate at each axle obviously

because this in most cases well the Imperial is full of empty in theory the bridge does not know if I have it and that this actually true unless you explicitly ask the app to modify it effective if you don’t do this the bridge doesn’t touch protective just send it out to be one or multiple other interfaces and internally pointers no handling of the empty checks on whatsoever just having all this code in Kiev just over bridge is kind of ridiculous so the solution for that problem is to make the bridge behave so when the bridge but the bridge calls Kiev and the app turns the bridge that the plaintiff has been notified this we calculate the IP checks on that was actually very easy and straightforward to do and I could basically delete all the old attacks on the code of the earth but now there’s the checks it is getting it to the way I explain why rich can use the offline actually even in the case of the bridge replacing the architects and we produce the offloading capabilities of such a benefit that but down the checks are of low I’ll quickly explain all that works as that pretty much everything mate in this century and / 10 years before has offloading capabilities at least for IT before the other stuff is really relevant anyway our stack has had I be offering apologies for 4 inches but I didn’t really check that this must be 2,000 ish but for a long time how does that work we had code that at the point that we had to check something cause but they always have to leave enough respect they have not been moved there was called to check whether our interface has checks on the capabilities if so there was no checks on being done just like set indicating that the packet needs taxonomy or the hard way ultimate if there were no checks on the capabilities on the inbound site the driver gets the offloading the checksum verification result from the hardware from the offloading engine and sets flags in the above header content error indicating whether the checks on was good or bad so if you see such a packet either as a flag saying the checks that was good the check so it was bad or it has normally spreads it doesn’t have any of these facts it’s obvious that we need to verify ourselves in software right the bridge the richest special case all over the network studies annoying as hell the root cause is the director behavior regarding taxes all other output Panthers take care of the techs on the bridge doesn’t because this theory should move packets but that theory is is the truly warlike has been for 450 years once the bridge is fixed all the special phasing all over the stack ago so I wanted to do that right why well what I did and I remove all the code and I think I become a minute I’ve and couple of days later no i didn’t know i didn’t because testing showed some the case was colored a testing portrait um yes we crossed paths to the bridge that ice covered in that the profs hack is regarding broadcast so a broadcast packets the bridge basically deals from a broad perspective might lead us with every other packet at some point but it has already decided where to send it is using the first list or something kind of randomly picks an interface for broadcast packet looks at those capability flanks whether we need to face this broad cut a check sonic apple or not and then just before sending it out to that arrow facebook quite a second this is broadcast this needs to go to all of your faces so with copy step Packers and puts it into the outbound juice from the other new faces now this is after you check something decisions that had has happened they are taking a short circuit well whatever you’re being super clever and taking the short circuit most of the time who really stupid this is what happened here

somewhere before he’s clever but actually this is not lever this fights back later or I was I did not fully track this down so I did not fix this get of course the entire journey is the anti things were complicated I didn’t want to get to Cyprus all right I put the special occasion back in and go on at some point this needs to be fixed status um with that I could remove all of my checks on heading from here British special amazing is still there basically the interface packet touches in British we always do check something that’s out there that’s all we didn’t for now that’s been the case for everybody face whatever it’s kind of sad since it is possible to use the confidence if there wasn’t that one passport so it’s on them someday it’ll be fixed I said this does not make me feel for performance because the episode 3 I did performance meadowlands but you can write it like benchmark as much as you want the difference is with and without a p-channel morning arms and the noise so doesn’t make a difference but what the Janos either because sort of a certified match them to follow and the IP checks on hunting is now in one place nicely self but we do not spread all of the stack the little joy disease Yankees excellent i think in pf was just plain wrong the that’s that’s that’s another sign of a much shorter nicole problem and in the government yeah place all of us vertaki f or TF hackers networks backpackers or BF hackers so we’re trying to do everything a key Evan I didn’t look at the red the network’s they all that much that has changed that has changed a lot but that’s the reason for this like you’re doing something to do Packard oh I’m a checksum does mention what we have to do something about it nobody really really happy a recheck something called in a speck on the right are back down so a lot of code written for nothing this should have been fixed in the bridge ten years ago before it was all this province coming to the protocol Jackson’s um this is mostly TCP ability I see if we have attract some other protocols are movin if they have some checks on there’s no uploading engine there’s no generic support and this is a man as a protocol or even application in a row that’s not going to be care about the the Oracle checksums will become a part of the appeal why if they would cover the TTL for example you would have to redo the checksum on each and every router and as mentioned it covers the entire panel so that is not free performance was also the router shouldn’t need to know about is it only gets over IP right so that will give a very violation the question why the protocol Jackson’s cover parts of the header of the IP header is a very global I can also learn it doesn’t make sense the recalculation here is potentially expensive not because of the math but because of tension all that data from small or whatever it might sit this is can’t really hear it’s just the same that we had for the architects on updated on the fly whatever Allah chose to and that’s the beauty nested cause to fix up 20 this is ugly ugly as hell and once again the ipd we should protest the more ugly vagina put now in this is master calls this verb even the lowest point is there longer chains in these long chains of nested check some clips of course it’s really easy to screw up it’s really easy even amazing compiler that loads because we’re dealing with words each and every parameter is a very right almost but ever touch the packet we have to do the texan fix on but they have to be this

perverse the check some pics on functional needs the ultimate the new words so that those were changed basically applies the Delta to the checks on this kind of clever but the word is not there I have together to you know workers of the snake it over for the Senators TV person this we will see an old an old hack from the BSG network stack but nowadays later than go out there all the SDP right network states use so called tropical control blocks to build for tcp making collections gets to track connections and to keep state unity being connectionist does not mean that it does not have steak the Luigi sockets need some kind of state right and that’s the interactive front of our control box in short i bought PCBs and they are being looked up using hash titles and openbsd the PCBs can get linked to the f states in that case you saved a lot of programs of course Venice rocket was opened by the insulin application TC penis creators of the template pce the easiest paste is the socket by the server side socket you bite and you listen on it so you already know the local I cameras and then you listen for any other connections it’s pretty much the same any other cases but that’s easier to follow the known Paris is the connection at this point are already still in the template pc and these are all really chicks now of course this is a hack there’s my strong language of all later when we accept the connection or that soccer player on the template PSD copied the other sites information like the remote computer s in support of whatever else is filled in and the checksum is just updated this checks are still is not good for anything really because now it only covers the little bit of the headaches right but it’s supposed to cover the entire payload the entire thing the couple of fields from the IP header and the tease me in her circles just like a resort is called the pseudo header checksum the packets that marked the paternal flag for I need checks on me it is passed on very late when the outbound call part is actually it’s late now at Euston urea we look at those tax on flex and we compare they seem to compare to the interface capabilities if the packet has abstract I need to the TC ipv4 types all amenities including for tech from and the interface can do it they’re done just pass it on set the flexor the author knows that it has to do it done if that it is excellent and don’t have any automotive capabilities obviously afternoon in software aside the old way this happened pretty early I changed this so that that basically were conducting available if it does not need any checks I mean first angle is forward because Martin has not been notified it’s obvious that bulletin a rising over here stuff the superhero Jackson that’s a hack that might have helped on the HP 30 relax the Nexus actually thought HP 300 slow on any half a motor system this is counterproductive this incremental operating and thousand make science more system will break the cache lines those systems and tiny caches in there all right water systems have big caches and big cash prize for that stuff sketch but there is no cost doing this at once because it was cheaper the incremental updating complicates things very very very conservative for most in the code so for still paying for for us for the code this hack some network interface card vendors eternal eyes this Hank and other so it’ll stay there are some some

chips for most the internal broken ones that require the zoo energizer to be there and there are flowing angel all takes care of the rest updates books to head of Kaiser everybody else implemented check some important capabilities did the right thing and just ignores what’s there they don’t require anything they can do a full check summary the writer checks out the protected zone there’s some that we filed a Texan zero that’s easy so the famous redirect to lock the most part this is one of the things that drove me to dive into this a packet that comes from the local host the scenario that that showed this is when you redirect something to localhost they reply packers of the proxy like relating you redirect to local laws particularly related us something and replace this packet coming from the local host has tips entire check sir from the template DC from the upper layers of the snake the template pc piece of that right then kiev rewrites because the packet had been in a record low post when we have to in the reply packet we have to exchange the source levels can be rich to the destination right so we write their faculty of course for this rewriting you have to update the Jackson but Kiev cannot know whether the packet has a full checks on or through hello Jackson so if you are changing something in the TCP header will have really update the checks on but the checks on there does not cover that part of the header yet reside is obvious which excels bro so desu three cases to consider the software engine doesn’t care it just doesn’t fully actually THX of now just business updating crap anymore the same hardware does care because it’s just ridiculous to text on the rise from the packet but but into the Bronco reliable hello Jackson to be there you’re obviously tuned because now they’re updating without objection so at a local mall surprise is still wrong so those Packers will be thrown away by illustration of people to check some so much the result of that is that they happen to say before go check some hopefully completely on these on these hearts it was well that’s just problem and that last of this is iain ege and BMX as the ninety percent of us 10 murmurs as what ERT checks on pretty much everything that’s made in this decade supports it not necessarily 586 but there’s a rail road anyway unfortunately there are many many many silicone box like back in the heart there we’ve seen the case where you turned UDP or TCP checks on the integral falling on the harpy will product or SPF patents yes specific prefixes would be group would cause corruption yes of course it is the community check something off engine has no point even touching the hosting a there’s a different phone call after just as des give me a key checks on the bridge was special and needed care basically the same problem and basically this has been fixed in the in one go with the fixes that info THX on however given for projects up is considerably more complicated because as three cases you have to consider that IP it’s easy uploaded or no offloading here it’s three it’s no offloading the superior types of required of building or full of loading they’re just care about the Samaritans so what a change we always want to work under the assumption that we have an automotive engine was almost everything is they pass if you’re hitting a path that does not have any offloading capabilities reconnaissance stupid never give this card or it’s an ID say it like that bill just called the software edge will basically emulate you have their of all the engines all this magic is in the new functions I am a 96 projects on out basically what they do is they check he the flash if there’s

check something needed click they either called the software engine or pass it on to the hardware for the right leg set to request the the check for good after 4i be informed there was a NP light checks on that space if you software engine in the seventh it didn’t have to be with her really 46 there was no such a thing so they had to write that from supposed to have occurred pain in the group the cars just finding the TCP header in an ipv6 packet is a nightmare before it’s easy its place offset these things you have to walk the entire extension headers so um to make better use of the obligations they are calling the projects of our functions very late not and they’re not pre calculating the checks on anymore we’re taking the decision that we have to check something or not very late but the circle of which interface the pack is going to go out which means that we can remove almost all other tracks on traffic from the stack I call this deep blue joining the stage totally it’s obvious for all the spaghetti code in the network stack is Bill joint where she goes she was adding one hang after the other it also can produce are the codes that road sorry this it’s any code is all over the spec instead of calling api’s oh wait a second I know the inside of this I have things like knowledge so I feel feel feel feel feel what do when you take you guys for so if you want to exchange to this thing you’re so that’s what we’re still dealing with now 20 years later so um that aside this part of the journey you can remove the checks on entering from basically all the stag and all over here just says it’s like this Banette said something and the checksum code is nicely self-contained and one spot and thus the right thing custom actually if you want to change anything there it’s one spot and not spaghetti all of this that I simply as a checks on to there is as far as i know that is a single bit of hardware single piece of hardware that has an icy checks on both holding engine it doesn’t quite seem to make sense anyway he takes to the doctor for most critical but we want we don’t want spaghetti and before I simply a nice coat for disability right so every model be the I simply inject some handling her very primary saying by 8am either for tcp and UDP we’ll just always end up for you so very good so what not a problem ICP arrows are interested first I simply errors rest of the Arab apples quote the disability but usually it is to be impacted might be something else but we consider the busy period because here they hold the packet they refer to so if you are signing setting a tcp I could say you’re trying to access one episode and then little media router has no route to that destination it sends the icing original back right and this i say p rho past parts of your original request it gets quoted so that you can match this ICP arrow and the tck the sessions that are most true now when we do that it’s kind of obvious that we don’t just have to rewrite the addresses from the outside we also have to rewrite the addresses in the quoted packet because otherwise the setting goals cannot match this ICP arrow to the tcp connection but when we rewrite the addresses well that call the method of course as a checksum in most cases it will be truncated so don’t care while trying a detector there’s no way to verify the check sir so only to be anything but that are very very small packets let go get Ron pages this Texan is very tribal fortunately there’s nothing at least I didn’t find anything we looked up and down there’s nothing that cares about the checks out in the quarter packets the exam for one great escape that’s a network testing thing that we used quite extensively because you can generate really fear factors with all that there is supposed to be that’s the only thing we know that

actually very feisty intentions but long um I have not implemented this but I think the taste is easy enough if you figure out that the packet is not faulted which is easy to figure out the course you could have to look at the FB header in the potent I can amen so there’s a length investment and actually the very nice you’ll find it’s not truncated then you could just recalculate the checks what we’re just calling the regular punches I said I haven’t done this because nothing cares anyway but we could we probably do it so consequences of GF the old way of handling the checksums was to update an incremental rewrite the new way is I don’t care I didn’t change in the packet sell flag it has to be recalculated if you update the prototype something effective already comes in from Jackson can be updated for the changes we do the checksum is still broken so that’s good but if you if you if you do all these changes is to just set the flag all this back at least check something and the entrance at the end rechecks on the record a checksum is mammoth you just fix this oops that’s not supposed to know so before rewriting Jackson they actually have two very fun unfortunately this is potentially expensive because it covers the entire play note so um this is the water restriction still now the question is is it worth it right I need to keep you updated way um why the code is so much easier and cleaner it’s not the spaghetti potion in words nice a certain thing so we want there’s still much much much more clean as possible afterwards and eventual in a really nice everything this fixes the redirect revocable obviously it requires it allows us to enable the protocol checks and offloading on the cards that we find them super hairy chest and it said this is foremost an egm VIII and IX which there is my percent of the total market so that’s mere Paris this unfortunately one case that suffers considerably from the strange but that’s my presentation compression wow I’ve never seen that before it’s japanese alphabet with magic for this almost always computer to generate up to escape it’s awesome except for this much in fact not my timer is up anyway yes the case that suffers considerably is when we are doing rewrite arresting that here I mean any form of net thiet rewriting the distance of destination ports or addresses all these like the temperature that we do net that do not have any offloading capabilities as mention this is very unlikely on recent hardware that of course happens because it has two very tight software every calculation software which means it has to fit all the payroll for memory I think that case doesn’t remember was one you have all available is everywhere to their reading micro national year if you’re doing anything more than just forwarding the cost of that anything more hides the cost of the checksum a good for anybody what we could do to fix that place you could check some just keep headers without touching the payload before PFS changes then we’ll all changes in checksum again basically apply the data it’s not straightforward right as possible on this one was made not to use to offload hindrance in that case and all unfortunately and the software check some headers and applied out of a is not real quick so we are hurting the optimal in case to support something or aged without their that you should just throw away ever again buy a Lincoln you actually is much as their for everything it becomes to be more common fighting so that’s the case I ready to optimize for them for old stuff if you care about performance and run spark

from 99 five servings for all so in micro benchmarking the lowest five to ten percent depending on the traffic pattern actually there’s one case that looks really really really evil there’s twenty five percent but this one here Michael benchmarking even smaller that’s its constructor case basically just all the name is I be 64 and also times I 36 of course how much news has on the specifics of the hop there as I keep mentioning cash is critical here so cache size makes a whole lot of difference the cash architecture legs a whole lot of difference how many their hands with passion that makes all of the difference so I just never in this lamont machine you are not getting the complete picture but obviously I don’t want to run this test for a horrible sheets like about my garage measurements we tested the all benchmarking or 1964 but this this is an architecture that almost guaranteed has these offloading capitalism’s but if you manage to construct the case there that’s that’s hurt badly it doesn’t matter in practice so I find these in five to ten percent in that case the advantages that all the other cases that actually matter cat very acceptable and boom and again by a consumer or a never bit of historic buy some used but will have even those two checks on offering also spent a hundred years so um performance changes from the checks are performed on a really recent Amy 64 system like a currency on cpu and doing people following we still cannot measure any difference unfortunately because otherwise if this one I explained earlier and the raw processing power in these is there is so fast that that the offloading doesn’t help them at all because limiting factors are not the CPU it’s Layton sees two devices and everything driving the very same tests on an older regime of time from adam i get about seven percent increase but 10 for something optimize that much jokes over the code that after the tea spaghetti code that paints the road for future improvement was not actually reading understandable and follower last at least it’s less code and it’s nicely coated it was for my keep mentioning this first the new place right it’s move one spots will be there substantial changes to that ever again is one spot actually to be 46 for Xena 215 and not not consuming your priors all over the net except because that is completely in and prone to be somewhat stable they can remove the stupid when I don’t check something from the kiss you and he stays completely because there’s no point if you see where the texts he’ll just be complete checks on the outer part that’s cheaper and of course the special phasing of the bridge has to die this will happen eventually if somebody wants to be known it be my guest I have to act on this couple things here because that’s the second major task we briefly touched the surface the checks on Hannity the snake is much more complicated than I ever imagined if you’re if you’re reading a bunch xlr8 that should be super simple then you start reading the code and keeping surprised I’ve been working the net which there for 10 years but it was not clear to be more unless the checks on tentacles just a nice gather that’s may what’s the first one actually understood what was going on with redirect no girls Falls very hard to figure out why this suddenly doesn’t program or he explained that to trust that the attackers more than once that took several times until we got what we was talking about and now that I finally heard on this I talk to you about this for 45 years and then start working on this a lot of testing very fine my code that helped a lot thanks for that a lot of testing help support came from my computer thank you special invention working because they have access to my speciality happened to the EC assistance

and the I tribution version of The Tonight checksum Headley was put in my camp thank you can I didn’t have to touch that was I actually started working at in 2010 that’s three years ago there were many many Agathon’s and conferences that I used to proceed on that there’s the last from Iceland in Slovenia Canada Germany to New Zealand for the people the maid persons organizing this and therefore the conference’s various legal age of these typologies we can you should go to all three of these conference they’re really awesome my talk any questions I could love you just ask them thank you let’s cheat on behalf of of the scientist who show respect to the burbs elbow out the gloves please leave us out of last year everybody in Japan doing something semi-official is wearing white gloves but are you are driving a taxi or just not with you soon you are wearing white gloves so I’m starting to giving a presentation I should wear white gloves I’m show respect to your culture which I really liked by the way thank you very much for your energy happy enough sorry I can’t hear you d we have any idea how many need support hardware ID checks and functional part if it was slower or if they need a mini dicks only soraa’s like there are so many suffer like in 1st citation sections so if the preservers are multiple now or I mean it’s about how security checks are necessary action cases I don’t know how often is it really necessary are neither hardware to do the suffrage excellent Oh almost there you’re almost always happy apparently the most common case theory to the software sex ollie is IP sick sorry if you’re doing a piece like the must go check something to pay load doesn’t matter yeah you touch all the table anybody because you’re crooked when the caches are not a dish with anyway so that’s the most typical case now otherwise almost everything has offloading there’s there’s the other case that’s kind of holiday after new software is the tunneling but then the tunneling is more expensive than the check something true yes oh yeah that’s not a performance critical performance relevant case the Holy performance relevant case is one evasion to do that and nothing else and more than any operating but that basically does not exist anymore it exists with some very very new hardware but the driver does not yet have the offloading support code but no that’s a mission usually and so far since the offloading didn’t really play out the motivation for people to add the offerings important the drivers was all that hide right now the last players the motivations thank you thank you