Image Processing is not What you Want to do Toward Automatic Image Analysis

My name is Mike Marsh, and I want to thank the organizers for letting me speak today. I am here with my colleague Nicolas Piche. Nicolas if you’re unmuted can you say anything Hello everybody, you’re lucky to have Mike doing most of the presentations today, so you won’t have to be annoyed by my strong accent, and anyway Mike is like now a Superstar LinkedIn from our webinars So you have to leave the work to great people, so that’s what I will do today. All right well, thank you Nicolas for that. So, that’s funny I can’t see the Q&A anymore. With that, so Nicolas is my colleague and he and I work together at ORS where we make some image processing software. We will be doing a demo on that tomorrow. Before I get into this provocative title, what you see on the screen is a title about the image processing, and turns out in this talk I think we have about 90 minutes, sorry 75 minutes slated for the presentation and questions and answers. We’re going to see a lot of deep learning, so I just want to tell you that now in case you weren’t interested in that, I’m sorry the title, there’s no, it’s not in the title on there is no aspect. But this will be talked quite a bit about deep learning. Now Nicola and I collaborated on this talk, but I’ll be giving most of it I may throw it to Nicolas, refer to him from time to time. So as you see the title is, image processing is not what you want to do toward automatic image analysis. We’re going to get into that provocative title in just a moment, I just do want to take a moment to mention on the sort of topics that we’re talking about today, we’ll also continue talking about. Right now we’re doing a live webinar series once a day, so if anyone’s interested in more content like this, about what we’re doing for image processing for electron micrographs, but also other scientific imaging capabilities, you can check out the Dragonfly Daily and there’s a YouTube site where you can go, you can visit this red link at the bottom if you want to see into the archive videos, I think Monday will be the 24th lesson, and then we’re doing one every day All right so I have this subversive title, here we are in an image processing workshop and I’m giving a talk titled image processing is not what you want to do. Don’t take us too seriously, don’t worry about that The thesis we are stating is that image processing may not always be what you want to do. And there may be better ways of getting what you want. So we will certainly stipulate image processing is very valuable and you should learn how to do it, so I’m very grateful for McMaster for organizing, McMaster and the CCE M group, for organizing this workshop I think it’s been fantastic so far and without a doubt you will be a better off of microscopists if you are well versed in image processing techniques But maybe sometimes you can get just as good results or even better without using image processing. Now I’m going to cheat and you’ll see what I mean when I define image processing here in a moment And for everything we’re saying there are actually many limitations and there are some caveats that we would apply to this thesis. So don’t take us too seriously. It is important at this point to make a distinction between image processing versus image analysis. And this has come up a little bit today, so for me image processing can span many things primarily for the focus of this talk, we’ll be talking about when you use image processing to correct artifacts. So if you have processing artifacts or if you have image noise or if you have a shadow gradients. There all sorts of experimental artifacts that may cause the image, the micrograph you collect, to not be representative of reality. There are also image alignment and registration and distortion artifacts, we won’t be talking about those so much, as we won’t be talking about image alignment, in the scope of this talk, but correcting artifacts that obscure proper image analysis. So image processing might be fixing your images, sharpening them, denoising them, strengthening them, enhancing their contrast. Whereas analysis is for the scope of my discussion, analysis is making quantitative findings from images. And that can be whether those images were unprocessed or they are processed. So I guess one of those should say processed So yes you can make quantitative images, and a question is image processing a necessary step down that path. Now we should embark on asking ourselves this question, what is the goal of scientific imaging? Now many many many times, we have a sample we don’t fully understand it, we want to understand its microstructure and, through this undirected discovery we prepare a sample and put it in the microscope. Often to satisfy our own curiosity, so that can be a goal of scientific imaging, to describe the microstructure or something that’s just not well understood. Now you may be like me and you found yourself in situations where you know the point you’re trying

to communicate in a story. You’re telling a story by writing a manuscript or presenting a poster or giving a talk, and you want an appropriate figure to illustrate the idea. Now in that case you might want a very high quality micrograph or figure to illustrate that So that’s a reason you might acquire an image in scientific imaging. Then we can consider different qualitative and quantitative analyses. So by qualitative analysis, maybe you’re a biologist and you want a visual description of where is the endoplasmic reticulum found in a cell. And you can take images and say, oh it turns out the rough ER is roughly in proximity to the nucleus, and so you can have that sort of qualitative description answered by scientific image Then we have quantitative measurements Maybe I take a large bunch of micrographs, maybe 100 micrographs, and I measure different points on the ER to the surface of the nucleus, and from 100 measurements I see that 90% of the ER is found within 500 nanometers of the nuclear membrane. Now I’m just making this up, don’t try and learn any cell biology from me. But the point is that’s a sort of simple quantitative analysis that you might answer from micrographs Then you might do deeper quantitative analysis with larger statistics from more than one image in multiple images So maybe in 10 different cells, we have fully segmented all of the pixels that we call ER and we are reporting a histogram of their distance from the nuclear membrane. So a full quantitative description not just a few measurements But a distribution of measurements. Or you might say we’ve observed mitochondria in neurons and we’ve observed mitochondria in HeLa cells and they have a different 3D topology and a different surface area to volume ratio So you might expect that you have situations where you want to establish quantitative differences. These are all different goals of scientific imaging Now once we’ve done imaging, why do we do image processing? Well, it’s a cold hard fact that imaging instruments introduce artifacts. What we want are images that are representative of reality and free from artifacts, and those artifacts may come from sample preparation, they may come from the imaging physics, or any number of factors. But the artifacts may interfere with our analysis because we may be interpreting artifact rather than interpreting the reality that the image is supposed to reflect. And artifact latent images are not as beautiful. We all like to show beautiful micrographs that don’t have noise and don’t have charging and don’t have shadows or curtain artifacts etc. So we’d like to have beautiful images. So we’re in a position where we want to do quantitative analysis and we’re worried that we’ll have garbage analysis if we try and process images that are filled with artifacts. So we’ll have garbage analysis with an analysis if we don’t fix our images with image processing Well if you believe this, that you will have garbage analysis, then I understand why you want to do image processing. But the thesis of today’s talk is, if you’re doing undirected discovery or you’re doing qualitative or even quantitative analysis, you won’t have to do image processing, maybe the best reason for doing image processing is when you want a beautiful micrograph to showcase. And so I want to convince you that you can often get quite a bit of quantitative value even from artifact latent images So computer scientists like to say, garbage in garbage out, and that’s true of most image processing algorithms, as most algorithms in general, that’s a bit harsh if we’re talking about someone’s micrographs And so I do not mean to convey that the micrographs I’m about to show you are garbage, they’re actually very valuable micrographs and that’s part of the point of this talk. So let’s just say they’re imperfect, so imperfect in imperfect out What we’re looking at here, here are some images shared by Nabil Bassim and his group there at McMaster University, and what we’re looking at here are some PFIB SEM images of some concrete samples. I know Nabil this is concrete and brick, I believe this is concrete. And you have pores in the sample and you have multiple phases, you have a matrix phase and another phase, and if you want to do quantitative analysis for example of the pores, you’re in a tough spot. In order for me to threshold to see the pores on the right you, would have to adjust the thresholding so high, that you would see, you would over label all of these pixels on the left which are just a little bit darker some sort of shadowing gradient. You also have artifacts that the plasma, the xenon plasma beam, introduces, and you also have this with a gallium focused ion beam. But in this case the plasma beam is introducing these curtain artifacts, and they to contribute to mislabeling, if I’m trying to label pores I don’t also want to count the curtain artifacts. These curtains are not a part of my intrinsic concrete microstructure, they are in fact an artifact of the imaging, or in this case that an artifact of the milling. Now here’s another example, this data set is just some purified HeLa – well I don’t know how I’d say

purified, but some perhaps HeLa cells embedded in resin, imaged by serial block based SEM – the images were collected by a Rachna Parwan and these data come, or these samples come from Dr. Xuejun Sun at the Cross Cancer Institute associated with the University of Alberta. Now these images are quite large, there I don’t know maybe four thousand by four thousand or six thousand by six thousand pixels. And so over the course of experiment a lot of data is collected, but in a rush to collect a lot of data, you don’t have time to dwell on the pixels and get a high signal-to-noise. So if you were trying to identify features in the image whether, those features are mitochondria or endoplasmic reticulum or whatever, if you try and threshold you’ll get signal everywhere because there is so much noise in the image. So a naive segmentation on images with high noise is going to be very difficult to interpret if you’re trying to understand what is the size distribution of my mitochondria or what is the volume of my nucleus or what is the spatial distribution of my ER with respect to the mitochondria Now here’s another example, we can look at where we’re dealing with artifacts that can confound our quantitative interpretation. And so I don’t know how well these movies are showing through WebEx, we had some problems yesterday, but you should see that these are going at about two or three frames per second and you can see us slicing through the data. So what we’re looking at here, this is not actually an electron micrograph, this is a synchrotron micro-CT reconstruction of some bone data from skeletal remains. And during the course of sample prep, this sample which actually has multiple phases, it has bone and it has superglue and it has air, but during the course of sample prep, I think someone introduced a file to file away part of the sample, and the file lists iron splinters behind which caused bad streaking artifacts in the reconstruction. So if I was trying to analyze this sample to understand bone shape and structure, I would be challenged to a high quality segmentation of the bone because the artifacts interfere with my pixel labeling ,with my image segmentation. So you see these bad artifacts. It’s important when you look at the images to try and decide what do you see, and what do you believe? So if you looked at this image, you could say with high confidence that this is the edge of the bone. I don’t know if you can see my mouse I’ll try turning on this pointer, nope didn’t work. So you can see the edge of the bone, you could trace it with your eye with high confidence. Now maybe if you tried segmenting the pores in the bone you would have to decide, okay some places I can segment the pores with high confidence and other places in the middle of the streak artifacts maybe I can’t believe it. But there are data in this image you can believe, even if they’re hard to segment with standard software algorithms. Now we could take standard software algorithms and apply them to our data to try and make it easier for us to process and then quantitatively interpret the images. But I submit there are consequences to image processing. In Christopher’s talk this morning, he specifically said there’s always going to be side effects, something’s always going to happen when you introduce in image filter, and so you have to be careful to understand what’s happening But what he also told us was the miracle of the visual system is that our eyes can pick out images from the noise We can pick out the features, and we’re going to build on that theme in this talk. So there are consequences to image image processing. So you have to ask yourself what are you seeing? So you’re no longer looking at the experimental image if you’ve done image processing You’re looking at a processed image, so now when you see a feature or you see an area without a feature, you have to do some interpretation, is the feature I’m looking at real? So is this pore actually in my sample? Is this an EM artifact? So is this actually a curtain artifact that I do not think is a pore, but is just a signal interpretation from imaging a spot where the ion beam has induced this this curtain like scratch on my sample? Or conversely if I’m looking an area that doesn’t have a pore, am I seeing a pore or maybe did I run a curtain correction routine that erased some real pores? So you have to interpret the presence and absence of features. Well, how good is your interpretation? And that’s going to depend on how well you understand the microstructure of your sample. Maybe I should say, well pores shouldn’t be collinear like this, I can tell that’s more likely to be a scratch induced by a plasma focused ion beam, or the physics of image formation in the electron microscope. If you understand that then you might start to understand, what signal you’re looking at, if it’s real or artifact. And finally do you understand the image processing algorithm you’re using? Because that’s going to introduce its own set of artifacts. Now there are side effects and, so you have to decide what you’re looking at is it a side effect or if you’re running a correction algorithm,

get it correct all of the artifacts you were targeting? So here we’re looking at an image and we may have actually seen this image in Christopher’s talk this morning. He introduced some discussion of processing and he also did a good job of setting up optimization and numerical solutions for solving problems. We will actually turn to that in a few minutes But this is an image from a paper, I found this paper on ResearchGate, where Chris and his student and Nabil, collaborated and looked at doing a really terrific job on correcting curtaining artifacts. Now in that paper, you’ll see this micrograph but you’ll also see this micrograph. And what they’ve done is they’ve looked at a large area micrograph with some curtain artifacts, then they’ve looked at their own algorithm for correcting it and then they’ve compared it to some of those in the literature. They’re not picking on these implementations, these are just implementations that do a good job and so they’re trying to do even better. So they’re comparing their solution to previous implementations. Now in the next panel, or in the next slide, we’re going to look at these three areas enlarged, and we’ll see them from all four views. So what we’re looking at here on the left, is a small area of an electron micrograph which suffers some artifacts What we see in the next three columns are different solutions for correcting those artifacts, both the one described in the paper and one described in a previous paper, and then one just one implemented with the Avizo software, the the one that Alex was telling us about over lunch today. Now when you look at this image, if you understand the microstructure of the sample and you understand the artifacts associated with ion beam milling, you might suppose that these streaks are not part of the texture of your concrete, but they are in fact an artifact. And will interfere with your interpretation. So you can run an algorithm, in here looks to be relatively free of streaks, here it also does a good job of making it free of streaks but it looks like maybe the texture across the grain might be different than in the source image And here’s another one, that does a good job of correcting the streaks but maybe there’s some residual in some parts of the field of view. Now there are going to be issues where it’s going to correct some areas and maybe not correct other areas. But it could also introduce side effects. So let’s look at this area. So there is a visual feature here which we might be inclined to believe is real and not an artifact. And then when we run the algorithm, I see that the pixels right here get a little bit brighter, white and they get even brighter in this algorithm, and still brighter in this algorithm. So if I’m interpreting this, I might see that side effect and misinterpreted as real signal, when in fact it wasn’t present in the raw electron micrograph at all. So when we run image processing algorithms we have to now ask ourselves a new question when looking at the processed image. Is this an artifact of the imaging, is an artifact of the image processing, or is it a real intrinsic microstructural feature of my sample? So those are some of the problems we deal with running image processing. So you have to ask yourself, what do you see, is it even possible to tell? So I submit that sometimes we’re better off when we only have two choices, is this feature real or is this an imaging artifact? It’s more complicated many times, when we have the third question is this a processing artifact? You have to know the image processing algorithm and what its behavior is and maybe how it works, to understand if what you’re looking at is an artifact. So we actually make this judgment call all the time, every day we look at micrographs. And we know or we make judgments of am I looking at a real feature or some sort of imaging physics artifact. And so that’s all we’re talking about right now So the statement Nicolas and I are making, you don’t want to do image processing, well why not? Well first by image processing, we mean filtering the image to make it look better. Well one reason you might not want to do this, is there many parameters to tune for some image filters, so do you have to become an expert,, do you have to run the whole image with one parameter or do you have to do a set of multiple parameters and tune them in different areas for different parts of the image? So that can be complicated and can be challenging to make sure you’re getting a uniform treatment over the whole image. And then of course, what we’ve been talking about in the last two slides, is there are side effects. So as Robert Heinlein tells us,, there’s no such thing as a free lunch, and it can very often be said that when you filter you lose information. Not always and it really depends on the algorithm of the filter. But very often that’s a true statement. So those are reasons you might not want to do image processing. Well what is the alternative to image processing? Well you could just do manual interpretation and a manual segmentation applied directly to the unprocessed images. This is completely not scalable You know you can’t do this. You collect 100 micrographs, you like 10,000 micrographs, you can’t interpret all of them, you’re going to need to rely on computational image analysis and input to the computational image analysis, if

it’s bad noisy images with artifacts, then you know you can’t rely on it. So but you can’t do manual interpretations as it doesn’t scale. So the alternative we’re proposing is, let’s say, deep manual segmentation. So it’ll still be guided by manual segmentation, but we’ll use a deep learning algorithm to solve the problem for us. So I’m cheating here, I’m not telling you not to use image processing I’m just telling you in many cases you can trade an image processing algorithm written by a scientist or engineer with a deep learning image processing algorithm trained by your manual segmentation. So I’m saying the alternative to image processing is more image processing, but specifically I’m going to tell you why we’re going to be using convolutional neural networks for that image processing, and the image processing here is designed to work the way your eyes and your visual cortex work, and have a similar capacity to see through and ignore the artifacts the same way you do with your visual perception. When you look at these micrographs you can see those scratches ,but you can immediately recognize them and tune them out. If you can train the deep learning algorithm to do the same thing, then you might expect it to get similar results to what your eyes and visual perception are doing. So I say here convolutional neural networks, and that sets us up for a good segment of this talk. What our convolutional neural networks, or for that matter how does deep learning work? This material I’ve presented a few times before, if you want to see this again there is it’s encapsulated in the series of tutorials actually from last week that we delivered, so the first one is called deep learning for imaging scientists, so you can go back and see that all of these are on YouTube. But we’re going to dive into some of those same slides and same content right now But if you’re interested you could do the follow up videos on denoising or image segmentation or super-resolution, etc. So what is deep learning? Well what we’re looking at in the lower right hand corner, I don’t know if the video is transmitting, but it should look like a video of President Obama speaking. In fact it’s a deep fake, it was designed and implemented by someone showing what deep learning could do to generate fake content. This is not an error as, Christopher tells us, we do not actually need to go and use an electron microscope to look at the microstructure of our chip ,it’s not that our chip is bad, it’s just that someone is sending us fake news. Now that’s one of the things you may learn about or see exposure of deep learning in the greater ecosystem Deep learning is everywhere And so you have a voice assistant, you may tell Alexa to buy something for you, or ask Alexa what tomorrow’s weather is. And it’s using deep learning to interpret the audio signal. You have applications on your phone that can translate menus for you, so you can order in a French restaurant, well maybe not you guys in Canada, you already speak French, but for myself. You have self-driving cars that can recognize when there is a pedestrian roadway or a bicyclist and adjust accordingly. You have lots of deep learning techniques that are applied to radiology and being applied to medical data. And those are all examples of deep learning. But today we’re going to talk about scientific image processing. So when we talk about image processing, I’m going to make a broad statement and we’re going to consider for example image filters for a moment that transform one input image into an output image. But also image segmentation that may transform one grayscale image into a segmented image And I’m just calling all image processing, and describing it in these terms or this pattern that involves a transform of some input into some output Now I want to consider these transforms as functions, maybe we can use the same fitting tools to fit these functions to our image data, the same way we can fit a line to scatter data. And this hearkens back to what Christopher was telling us about optimization If you collect coordinate data, so in this case I have data plotted as X&Y. If you collect coordinate data ,you might observe the coordinate data and say the data are mostly collinear, I want to have a function that describes the relationship between those two. And if you use a simple function, you might understand all the methods, just like if you have a simple physics equations you might do optimization and try and understand the physics of the system by finding the parameters of your equation Here the parameters of our equation are quite simply M and B, if I’m trying to find a line defined by y equals M times X plus B. But we will see when we take this to much larger problems with many more variables, we’re no longer trying to understand the physics or the math, we’re just trying to have a mathematical function that we can use to transform input data into output data, noisy data into denoise data, or grayscale data into segmented data. When you have this

solution and you’re trying to perform this optimization and find this function that fits your data, you also have to have a cost function. So if I make a guess that I want to use this line, I need to evaluate how bad is that guess And so in this case the cost function is, for every observed point how far away is the predicted point? So I could take these differences from the predicted point versus the observed point and come up with some cost function that says, this line is a good fit. And if the line went this way, those deviations would be far and the cost function would be high So I have a cost function then I have an optimizer, that works and samples this this two parameter, space M plus B, until it finds a function that has a very low cost function, or I should say finds an equation of a line with these two coefficients M and B, such that the cost function is minimized. Now that’s the problem of fitting a line to coordinate data points Now in image processing, I need a function that can map this image into this image. So I’m observing not coordinate data, but image data. So this begs the question, what function are we trying to fit? This is not y equals mX plus b, this is something altogether different. And once I have a function that map’s the input to the output ,what is my cost function? And then if I have a cost function and I try and optimize all the parameters, what are those parameters, what are those coefficients I must fit? And maybe even another question, what optimizer will I use to fit those coefficients? Well the primary answer to this question, what function are we trying to fit, is going to be neural network. So I’m going to spend a few minutes talking about neural networks Before I do I want to make this distinction you. See deep learning a lot in the literature, a lot in popular science, you also see machine learning. People that use it precisely, will use it consistently because, listen with this Venn diagram I’ve drawn here. So artificial intelligence is this broad scoped idea and it can mean anything when the computer seems to be exhibiting intelligent behavior. If I play a game of chess against a computer and it seems to know a good move, that computer algorithm could be considered artificial intelligence. Within the scope of artificial intelligence there’s this whole class of algorithms or mathematical models that we call machine learning. And those are computational systems whose performance improves with the more examples to learn from. And there’s a whole host of those. One specific type of machine learning algorithm is artificial neural networks And what we’ll see is that if you have an artificial neural network with multiple layers we call that deep learning So the takeaway on this slide is deep learning is just one type of machine learning among many. And machine learning and deep learning collectively fit in the scope of this artificial intelligence. You’ll see the same slide recreated and represented all across the internet. This is the version of the slide from Nvidia, where they describe artificial intelligence with an onset in the 1950s, and then machine learning that you know kept the spam out of our inboxes, and then deep learning where you could tell a tell a program to watch videos on YouTube, and now say that’s a cat or that’s not a cat, even though you don’t program it and tell it cats have whiskers or cats have pointy ears, the whole idea is it learns from examples. And we’ll see that in a few minutes, not cats, but we’ll see it with scientific images. In the scope of evaluating deep learning, as compared to other machine learning techniques, this is a generalized diagram that we use, which is to say, the performance of a lot of machine learning techniques can get very good with a little bit of training data, but they often hit some sort of limitation. They did some sort of asymptote on this performance curve. And to the contrary, we find that deep learning techniques, they don’t learn as quickly, they need more training data, but if the network is deep enough they just seem to keep learning more and more and more. Now this is a gross generalization, but the point is deep learning techniques in general require more training data than machine learning and they outperform the performance of other machine learning techniques. So that’s why we turn to deep learning. In tomorrow’s talk we’ll actually do a hands-on demo, and we’ll talk a little bit about random forest machine learning techniques and deep learning techniques and how you can put them together. But the point of today’s talk is really about deep learning and solving these problems, and this is why deep learning, why we like to talk about deep learning because it outstrips the performance of other machine learning techniques Now in order for us to understand deep learning, we need to introduce the concept of an artificial neuron. And this was described a half-century ago. But suppose for a moment that you go out and you walk around the neighborhood and you make observations of the animals you see And you build a table of data and you note the height of the animal and whether it comes when you call it or it ignores you when you call its name or whether it sleeps all day. You just record these on a table. If you then had some sort of function, and this could just be adding up all of these values with some sort of coefficients or some sort of weights. And then maybe changing the output of the function in this case, maybe normalizing it to between zero and one. There might be a set of weights, such

that all of the animals that were cats would have a value close to one and all the value all the rows that our dogs might have a value close to zero. And so you might just draw a line somewhere and be able to say with pretty good prediction, this simple mathematical function of adding up the inputs and then normalizing them, might be a predictor of whether an animal is a cat or a dog. So that’s an artificial neuron or sometimes called the perceptron. Now what we want to do is map this onto image data. And an image is not a bunch of measurements of cat weight and cat height or animal weight and animal obedience. But we can think of an image as a bunch of pixel intensity measurements. So this is a handwritten nine from MNIST data set, so it’s a it’s an available data set. And this is a handwritten nine, where the pixels that are bright white or the pixels that have the most ink, and the pixels have the least ink are black. And so you could imagine this not as five or six measurements of animals, but actually 784 pixels measurements. So you could consider the input to some neuron just being 784 inputs, 784 pixels intensities But before we get to the image let’s consider another example. In this case suppose someone collects the width and the height of some pieces of furniture. And so we want to understand for each piece of furniture, can I predict based on width and height, is it a chair or a bed or a table? Well in this case instead of building a neuron that says, it’s a zero for a dog and a one for a cat, we could have a neuron that says, when I think it’s a bed I’ll signal one, and when I think it’s not a bed I’ll signal zero And another neuron that says, when I think it’s a table I’ll signal one, etc etc. And if you have three neurons and each one of them takes the same inputs, the height and width of the measurements, then you can look at this output layer, and say well maybe whichever one is highest, is the one my network is predicting. So here we actually have three neurons, in this single layer of neurons each getting the same inputs. So this is called a single layer perceptron And it has some predictive power, it can help you tell if am I looking at a bed or a table or a chair. Now if we return to that image example, and I have 784 pixel intensity inputs,, and I just plug them all in to nine different neurons or sorry 10 different neurons, turns out no. This does not have predictive power, just having a neuron with a bunch of weighting coefficients for each of the different pixel intensities is not enough to predict, is it a zero or is it a one. A single layer perceptron lacks that capability. But there is this concept that was expanded in the 1980s, called the multi-layer perceptron, so instead of one layer of outputs 0 through 9, or is it a table or is it a chair, we can have our output neurons. And buffered between those output neurons and our input signals could be another layer of neurons, let us say pre-processing the data somehow. And then passing that on. And so this neuron is actually not taking the input of the table or the pixel data, but is actually taking whatever this has done in some sort of weighting function. Now whenever you have this architecture where you have one or more hidden layers inside the network, this artificial neural network is now deep learning, or the feed-forward network or the multi-layer perceptron. So really one or more hidden layers is the hallmark of deep learning And these networks are very very powerful far more powerful than the single layer perceptron. So if we return to our image tasks, and suppose I wish to classify, is this handwritten digit a nine? We could build a network to do this. A multi-layer network to do this. And this image and the ones in the next several slides are all taken from Grant Sanderson’s video series. So if you want to spend more than the next ten minutes that I’m going to talk about this or maybe only five minutes, you can really dive into how neural networks work by looking at Grants Sanderson’s three brown one blue channel. He’s a fantastic educator at making math tractable for people. And this series of presentations on YouTube, really does justice to how neural networks work. But I’m going to borrow a few of his examples and slides So if we want to map this image onto a neural network, so we have 784 inputs. And the task is, determine if this number is a 0, a 1, a 2, etc. We’d like it to light up as a 7, if it’s a 7, and have all the other neurons return something close to zero. So in this example, grant constructs a network that has 784 inputs and 2 hidden layers of 16 neurons. 16 is actually arbitrary, and you might get good success with just eight neurons in each layer, there are different experiments you could do, but for this he uses a network with 16 neurons in each layer. Now we have 784 input pixels, and we have 10 outputs, and these two layers of 16 neurons, they all have weights, they also have a function that I mentioned earlier, they also have a bias, which I did not mention, and in total this network has 13,000 coefficients to fit

Unlike fitting a line which only has two coefficients, slope and intercept. When you put slope and intercept ,you really know the mathematical description of the function. Here I could give you a table of these 13000 weights and biases, but that’s not really going to help you understand how the eye decodes this number. Those weights are not going to give you something meaningful, but if the network is able to give you a result, then that whole transformation that function itself is meaning, even if you don’t understand how it works. As I say each neuron in the first layer takes 784 inputs, has some sort of function, some sort of coefficients and weights, and then sort of an offset that helps to determine whether this neuron gets a strong enough signal to process. So if I build a network, and I want it to be predictive of is it a 0, a 1, or 2, etc, I really have to teach the network. I have to somehow define all of these weights, you’re not going to tune them by hand, so machine learning in this case or deep learning specifically is finding the right weights and biases for this network to work. And this comes back into play where we have to consider the cost function. So, suppose I give the network a three it’s going to make some prediction and we can evaluate how good or bad the prediction is. Now to start the first time we present the network with a number, we’ll just plug in random weights and biases. So we’ll put garbage in, that’s a garbage network, and we should expect just utter trash to come out. So if we put a three in and then we look at the nine, sorry the ten neurons on the last layer, well some have a low value some have a high value, so it says maybe this is a three, I think more likely it’s an eight, this is pure random, because these are random weights. But I know with the ground truth is for this, I know it’s a three, so I could say this should be a three, with the score of one and every other neuron should be lighting up with a score of zero. And so I can say the error or the loss function or sometimes called the cost function, we could just say let’s take the square of the difference, so 0.43 minus zero squared ,and then sum it up over everyone So that gives us a cost function for, how wrong is the last layer, how wrong is the result for this network. Well what I can do is I can put a three through the network, and I get trash out, but I can make a note of the cost function. Then in my network I can say, if this number wasn’t high enough maybe some of the weights leading into that neuron should be higher. And if this number was too high, maybe some of the weights coming into that neuron should be lower. You can make adjustments in those weights, and then you can go back another layer in the network, and make adjustments in the weights leading into those neurons, and so you can back propagate and nudge the weights and write in the right direction Then you can repeat, present another number note the cost function, adjust the weights, and the network learns by adjusting the weights. How you adjust the weights is the job of the optimizer function. So it notes the cost function and it has a mechanism for adjusting the weights in this 13,000 parameter space Now this task of mapping an image onto a neural network, maybe we should ask the question, as Grant does in his video, why would we even expect a network with this topology to work? And there’s some reason to think it might work. So suppose you’re trying to recognize the number nine. You might say, well the nine, I think it’s the combination of a pattern where I see the line pattern and a loop pattern. And so maybe if I had in this layer of neurons, a neuron that recognizes loops near the top of the image, and another neuron that recognizes lines in this field of view Then I could have high weights coming out of those feeding into the nine neuron, and maybe if I was trying to guess an eight I would have a high weight coming out of this loop neuron feeding into the eight, etc. And maybe the loop neuron, this pattern is really the aggregation of different features. So for example all of these different line segments could be detected in the feature detection layer, and then help identify a pattern, and then the combination patterns could help identify a digit. That’s a reasonable way this network could work. Well we can look at this layer of neurons and we can see does this neuron detect a line segment, and does this neuron detect another line segment? The features that lead into these patterns. Can we decompose this pattern into these features, and does the network do that? So it could look like this. In reality, if we examine this first layer of neurons in this network, what we’ll see is not the features we expect, but a completely different basis set. So this is not at all what we expected, but it turns out there are a nearly infinite number of combinations of the feature detectors we could aggregate here, that could be linearly combined to detect patterns, that could be linearly combined to detect numbers. So we don’t get the basis set that we expect, but we do get a basis that is discriminative. It does enable this network to see the difference. In fact with this network, as trained here, you get 96 to 98 percent of

accuracy If however you introduce a layer before this layer of neurons, where you specifically allow the network to have convolutional neurons, these convolutional neurons are specifically designed to detect those sorts of features, like edges, and the features the detected are not programmed, they’re actually learned over the course of the learning. So if you introduce a layer right there, then the accuracy the network goes up to nearly 100%. And whenever it misses, they’re really borderline cases that some humans would miss. So this network example describes how an artificial neural network works, if we introduced a convolutional layer it would be a convolutional neural network. Now I was just describing how to detect numbers, but in our case instead of detecting loops or lines, we actually want to classify an image. We want to label the individual pixels as, this is a poor in my concrete, or this is bone and not bone, or this is a mitochondrion in my hela cell and not mitochondria, because I want to do quantitative analysis. I want to label the image. And so we don’t actually do a loop detection. So instead of having ten outputs on the last layer, we actually have the same number of outputs that we have inputs. And we use something called an autoencoder network architecture. And this was first, this new artificial neural network, would convolution for segmentation or what the literature sometimes called semantic image segmentation. This was first described by workers at the University of Freiburg in 2015. To amazing success. and so in today’s lunch talk by Alex, he mentioned that the network he was using to do image segmentation was U-Net. So this is not the only network, there many networks you can use. So the network is a very nice mathematical model, it’s similar to biological neural networks, in the network learns what is it’s important about an image the same way you learned what’s important to be able to differentiate someone’s face from another person face, or am I looking at a bicycle or a car. So neural networks can learn all of those models. It’s also true that the more training, the more robust the network, just like in your own brain, the more you study the better you get at something And it’s also true that training is more computer tested than inference If you try and learn anatomy, it may take you a long time to learn an anatomy, but when you go to recall anatomy that is applied in the network, you can usually summon the answer in milliseconds. There are dissimilarities, we won’t get into this but the artificial movement works obviously have differences than biological neural networks. And as also I didn’t talk much today, but there’s something called the activation function, which in some ways belong in some ways is analogous to the action potential of the biological neuron. So in our software we tried to solve this and we rolled this out a few years ago, and of course we saw it at lunch today too. Deep learning based image segmentation, I’ll just talk about what we do at ORS for the Dragonfly software. So we use tensor flow, just like almost everyone else, there are a number of different frameworks that people can use for deep learning Pi Torches is much more popular now, but we use the TensorFlow and Keras family of tools. And this is just an internet plot showing how much people comment and contribute on different libraries trying to help you understand how popular they are. So TensorFlow is a framework we use, and in Dragonfly we build our code directly on top of Keras, which is an abstraction that can sit on top of TensorFlow, or the University of Montreal theano, which has been now adopted by someone else. Or the Microsoft cognitive toolkit, but in our case we’re just using the Keras TensorFlow stack So we try and create a system where you can use something very much like an image filter, that is it just transforms an input into an output. And you can reuse those filters, you can train them on new data, you can share those filters with other people with what we call the infinite toolbox, which is our App Store But the workflow here, is the user paints some representative slices or areas of an image and says, if this I believe is a pore, and by painting a pore and by not painting a curtain, you give the network examples of what is a pore and what is not a pore, and it learns to see through the curtain artifacts. So the user can configure a model for training. Train a model and then expect it to segment the images in the rest of your data set. So it learns on a few examples and then it can extrapolate to your other data. So the challenge here for us was just to make it useful, make it flexible, make it easy and portable. So that’s all just software engineering on our part. So I won’t get into that, we’ll talk a little bit about it tomorrow. So I’ll go through a few slides We’ve seen success with deep learning in many many different ways. Here’s an X-ray micrograph reconstruction of denim fabric, where you have large fibers and you have small fibers, and you can label them manually quite easily, you can manually label five slices in about five minutes, once you know how to use the tools, and you can train the neural network to do this differentiation for you. And the neural network can take this grayscale stack, and it can label the small fibers and label the big fibers, and now you can use digital software to hide the big fibers,

and see the microarchitecture on the inside, in a way that’s really hard to interpret just by looking at the raw slices. You can do harder problems, like I want to know where the crack is because it has different consequences than where the pores, are even though it’s the exact same signal in this Z-ray micro CT reconstruction of iron pellets. But the deep learning model can learn to do the differentiation, and this work was published this spring. I don’t know if these videos show very well, but this is a micro CT reconstruction of a piece of chocolate, so this is KitKat. There are many different layers in the chocolate, there are a little chocolate chip phases embedded inside another chocolate matrix, and then a chocolate waffle phase. And what you want to do is label the different phases, and so you can use deep learning to see the differences, and then go through the different materials. So it can work on large 3D datasets with complicated microstructure, a micro CT data set of a ceramic matrix composite that has very low grayscale contrast, but you can segment the images and then do interpretation in axial longitudinal sections. This just shows you the difference of the segmentation. So this is automatic segmentation of the data, directly by deep learning methods. Now another video, just some eye candy, showing you, you know, after you’ve processed the data you may want to render it and make a movie. So this is rendered and dragonfly to help communicate the results to colleagues, so they can see what the microstructure is, what the exterior barrier coating looks like, what the matrix in blue looks like, what the silicon carbide fibers and orange look like, etc, Now you can also do very very subtle features, there are there are small cracks in the data that can automatically be detected by a deep learning network. If you can see it with the eye you can train a deep learning network. And then you can run it in 3D see where those cracks are in the 3D data. You can also do it of course with electron micrograph, since that’s what we’re talking about today. Here’s automatic segmentation of rat neurons, these data, oh I’m terribly sorry to Angel – I thought the attribution was here, these data were collected by Angel Paradas at FDA in Arkansas. And these data are six or seven years old, but this is all done with automatic segmentation, here to show you separating labeling the mitochondria, and the nucleus, and the cell body, as differentiated from all these surrounding processes, which may not be interested, so they were just not trained as part of the model. Here’s a closer look at sort of what that segmentation looks like. Now there are a number of different network topologies, we heard about U-Net. There are always new contributions to the literature of different network models that can have useful solutions for processing data. Here I am using processing again, so we are doing image processing, we’re just trading maybe human engineered algorithms for learned algorithms done in the way of artificial neural networks. So a number of these network topologies are used for semantic segmentation, some are used for super resolution or object detection. The ones listed here are just the ones that we’ve implemented in our software for other people to use. But our software we make it possible for anyone to implement their own neural network architecture, if you’re deeply steeped in the literature, and you know what you’re doing you can make your own models and tune them, etc So it can also be used besides for image segmentation, it can process and denoise images if you want it to, although I’m sort of making the case today that you don’t want to denoise the image. Here’s some FIB SEM micrographs from Keana Scott at NIST of Sea elegans and what it looks like if you might denoise those images. You can also do something called instant segmentation, where you’re trying to label the different grains based on some ground truth trained network. So not just segmentation and denoising and instant segmentation, but you could do super resolution methods. So if you look the, image on the right is sharpened, and you can sharpen this by training a neural network. So there’s a lot of this in the literature lately. So this is a nice white paper we found on the internet, describing how Ryan Armstrong and his colleagues at the University of New South Wales do it. They even publish their data, so other people could reproduce it. So we just spent maybe one afternoon reproduce the architecture described in the paper, use the exact same data and we’re able to reconstruct. So here’s the high-resolution ground truth image and here’s the low res image, and here’s what you’re able to capture. So you don’t get all of the texture, but you get the surfaces at a much higher resolution than you do from the low res images. So I’m not going to go through this slide, just telling you about where we’ve gone and continue to improve in our augment, our deep learning. But let’s return to those examples we looked at the top of the talk. We looked at the concrete sample, and you can train a neural network to see past the curtain artifacts so these are just some figures my colleague had on a notes page. So there’s a manuscript in preparation right now, on looking at these samples But you can see I think in this case, labeled in pink some pores and some area of another phase labeled in blue, and then the main matrix phase left unlabeled. And on the right are different neural network models, and here they are reproducing the validation data. So if you give it sufficient training examples,

where you’ve hand painted and say this is what my eyes believe are a meaningful phase and this is what my eyes believe are not meaningful, then you can train the network to reproduce it. And here’s another example, in a different area the image, again with the magenta for what the microscopists says he believes are pores, and in in red what the microscopists believes are that secondary phase. And then different neural network models, in this case and working in 3D. And here I think in 2D to process the data. Should also note that you can feed as input to the neural network multiple image channels, so if you have the secondary electron and the backscattered electron, or you have a CBS detector that gives you multiple signals, you could pipe all of those in as input to your neural network model. So you can get quantitative information out, even on int data that has not been smooth or curtain corrected or processed. But here we’re looking at that noisy HeLa cell with the thresholding on the left, and on the right are all the labeled pixels of the mitochondria. If you look, it makes some mistakes and label some things in the nucleus, which if you look very closely you could mistake for a mitochondrion, except for the fact that you know they shouldn’t be in the nucleus. But this segmentation is so free from noise that you could just quickly go through and just manually subtract out the ones that appear in the nucleus, it would take almost no time at all to process this And so here we are processing not some filtered data or denoise data, which may have had artifacts introduced but process it directly on the noisy data Now the other example I showed you was the bone with the streak artifacts where you have these bright streaks that go through the data. But if you’re just interested in constructing the surface of the bone, it can see through those streak artifacts the same way you or I can. So you can train a model and this is just a one day experiment to paint three slices and then train a model for an hour and a half. So I’m going to wrap it up now. The idea here is using deep learning to work directly on, let’s say the dirty images, because you might have a better interpretation of the dirty images than you do with processed images So when you’re tackling scientific images, you may consider image processing, you may find a suitable routine ,you can learn how to use it and tune the parameters. If you’re unlucky you may never get acceptable results for the whole image or the whole image stack, or if you’re really unlucky there may be no image processing routine that really meets the needs for your particular artifact. Or you can proceed without image processing. The data may look bad, maybe you might want to call them garbage, but you don’t have to throw it out, it’s not really garbage. If you look at it and your eye can see meaningful findings. Then you can train a deep learning model to do that segmentation and find those same features, despite the artifacts that are present in the data. Now you could also use image processing and deep learning So if I have one of the frames from that HeLa cell, and I have a noisy mitochondrion. If I want to prepare training data, if I use a denoising algorithm, that is I do image processing to smooth the image, and I get something like this, it might be easier for me to label where the mitochondrion is in the denoised image. I could then take this same label and overlay it on the noisy image. I could after I perform manual segmentation on the denoise damage, I could then now train a deep learning model, which is what I did in this case, training a deep learning model to work directly on the noisy image and try and find the mitochondria in the noisy image And it has the advantage of its working on all of this data for better or worse, rather than working on this data which maybe don’t know what artifacts were introduced by the image processing. So when you train in this way, where you’re using this to help you segment, you are implicitly trusting the denoising filter, because it’s guided your manual segmentation. So you have to decide if you can trust it. Now you could use deep learning and train directly on the denoise the data to get your answer. Well now you’re explicitly trusting this image processing routine, because if any signal that was here and lost in the conversion of this to this, it’s no longer available for the deep learning algorithm to assess. So it has to work only on this, which has any trade-offs and corrupted data that were introduced by the image processing algorithm. It might train faster, maybe it’ll train better, this is really an unexplored area of how much better or worse or the results be with these different pipelines. Now there are some limitations to deep learning, it’s not really well suited when your images have really great brightness and contrast variability across slices. So you might want to use an image processing algorithm to normalize them. Or just give it lots of training data, where you have representative slices that are bright and represented slices that are dark. Now in our limited tests so far, it’s also not really good when the image has a strong shadow gradient. You know you sometimes have signal occlusion caused by the walls with FIB SEM trench. If you know what I’m talking about, you may have trouble using a deep learning model to process those images directly. We’re going to keep working on it and see if we can come up with a good deep learning model that’ll tackle that problem, so you don’t have to use separate shadow correction algorithms. So this is my last

slide, I’m wrapping up, and we have maybe maybe 16 or 17 minutes for questions, we’ll see if there are questions or send the last talk of the day so it doesn’t matter if we end a little early. But is the last slide. So I want you to be cautious when using image filters before you do your analysis. So be on the lookout for side effects, and take care of when interpreting the processed images, whether your you’re interpreting them visually or algorithmically, you know, that curtain correction algorithm could introduce a small band that’s just a little bit brighter white than the neighboring pixel, but if it’s an artifact you want to know that’s an artifact of the image processing algorithm and not real microstructure signal from your electron micrograph I want you to consider deep learning approaches, and they can be fully informed by your own visual interpretation, so let the deep learning model learn to see whatever it is you see in the images. And it’s just extending your manual interpretation, it’s just using a copy of you in the computer, a neural network that’s a copy or a small facsimile of the part of neural network you’re using in your own brains to process the data. I would also say don’t reach, don’t expect that you’re going to get more quantitative data then you can reasonably see. If I look in my image and I can’t tell if this is a pore or if it’s an artifact, then I’m not going to expect a deep learning algorithm to see it. But there are some exceptions. If you look at the whole field of cryo-EM from the 80s and 90s, you can have all of these individual EM images that are so terribly noisy that you don’t see any features except when you align them and then add them up and add the signal and noise. So sometimes there’s a lot of signal in there, even if you don’t see it, but sometimes you only get that signal out by adding it together collectively. But that’s for a different talk. So that’s all I’m going to say today, I’m going to open it up for questions. But this is the takeaway, is consider using deep learning directly on unprocessed images, and let us know how it goes. We think you’ll find there’s a lot of times when you won’t have to result to image processing, and you’ll have a more informed interpretation of the images that haven’t had any artifacts or trade-offs introduced. So I’ll stop there and I’m going to pull up the questions and answers panel Looks like there was one but you ended up answering it a few minutes later in your slides. Yeah, sorry, do you have more to say about that one? Well is this is this the last one in the thread from Benjamin Qi, are there a few? wait So other models except for U-Net to identify particles in different categories, such as image net and ResNet? So image net is a classification library, for saying am I looking at a bicycle, or a person, or a car So that’s object detection, rather than semantic segmentation. We have that in dragonfly too, at least in the next release, but it’s not really the same as labeling the pixels which is often what you want to do. If you just want to count the mitochondria then you might use an algorithm like that. ResNet is another segmentation network. We’ve not had as much success with it for scientific image processing, although we’ve implemented and tried it in the lab. For denoising we use U-Net, but we also use other network architectures. Can you share the links on your slides and online courses? Certainly, so we’ll get those slides, I guess I’ll get those to Samantha this evening, and then she can and you can get them to everybody. The next question, are their comprehensive data sets and other models except for U-Net to identify particles? I guess that’s just restating of the same question. Okay. And that’s it there are no more questions? There was one that was sent to me privately. It was, how much manual segmentation is required to train a model reasonably? Yeah, that’s a good question. In the in the case of these data, I could probably answer some of them. So I trained, I manually painted three slices, I spent about maybe 25 to 35 minutes preparing manual data for this model. In the case of the HeLa cells, I painted three slices, I probably spent 6 or 7 minutes on each slice. Alex talked today at lunch about the fact that you can use data augmentation, and I don’t know if we’ll talk about that during tomorrow’s tutorial, but that is something you can do here as well. So the amount of training data you’re providing as input can be augmented, that is, you can make some of your training data count twice or three times. So if you have a frame that shows a mitochondrion, and you can also present that same frame, but maybe it could be flipped horizontally, or flipped vertically, or rotated, so the neural network gets to see it in a slightly different view and learn to recognize it from that view. So it’s counting twice, you can also add Gaussian noise or local distortions. There are other ways you can augment the data. What we’ll look at tomorrow in the tutorial on what we call the segmentation wizard in dragonfly, excuse me, is we’ll look at a way you can get by with little training data. And

we’ll look at an iterative process where you provide a little bit of data and you get a decent model, and use it to predict on a large slice, then you can go and clean up that slice and it takes a lot less time to clean up a modestly predicted slice than to paint an entire slice from scratch. So you can bootstrap your way into getting more and more training data very quickly in this iterative process that we introduced with the segmentation wizard Okay, another question, came in privately How do you know if your deep learning model is correct? Yeah, well that’s a fantastic question And most of the time there’s no answer And it’s also the same case with the image processing algorithms, how do you know that the processed image is correct? What you want, and sometimes this is possible experimentally, what you want is a ground truth. So when it comes to image segmentation here, we often don’t have a ground truth. We use the experts of a microscopist, the expert eye of a microscopist or a cell biologist or material scientist, to label the different materials that he or she recognizes and that ground truth There is a scoring function, you can run it on validation data and see how well your network performs. But you don’t know if it’s correct, you just know if it matches your manually curated data. You can compliance to another image as your ground truth. So if you’re trying to train a denoising network, you could capture one micrograph on the microscope for one second and another micrograph for 25 seconds, and then you have a very low noise image you can use as a ground truth. So you sometimes can establish a ground truth, but basically the question, how do you know if it’s correct? You can use different scores, there are different scoring metrics. And in our software we allow you to use a host of different scoring metrics to compare your accuracy to your validation data. Your validation data can be prepared automatically by extracting out some tiles of your image or it can be prepared automatically, sorry you can provide a separate data set, so you could if you could provide three data sets and use these two for training and use this one for validation So you can partition the data manually or can be partitioned for validation automatically. There is a question in the Q&A panel, about how easy is it to add more training data to your neural network from different data sets? In our framework, you set up the input images and the target images. So maybe I have a stack of 100 slices, and I’ve painted slice seven and slice twelve, and I’ll put that as input. My input image and my segmentation, and you’ll just create a little mask that says only pay attention to slice seven and slice 12, but then if you have three other data sets from three other experiments you can just add those as well. So you can actually stuff in multiple pairs of training data at the same time, and it will take them all and shuffle them all and use them all with equal weight. The next question is, can we take the process data from a deep learning algorithm and use that as validation data after minimal correction to improve the accuracy? If yes, how affected is it? Well, I mean I wouldn’t take the output and use it as validation for the algorithm itself. What you want to do is, you want to set aside some of your ground truth data and use that to score your network, both the final output, but also score the success of the network as it’s going through its training iterations. So I wouldn’t set it aside, I wouldn’t take the result and use that as validation data. But you can take a prediction and then manually clean it up all you want and say now I have more ground truth, because yes it was you’re labeling the data was accelerated by using the output of a filter, or in this case a neural network model, but you’ve manually curated it, so now it’s ground truth. There’s another question, with respect to checking accuracy of deep learning, oh maybe it’s an answer from Alex? With respect to checking accuracy of deep learning results,, you can check the accuracy on validation data it may be more interesting to consider a loss function as the model trains for more epochs. So I don’t know how it behaves in Avizo 2D or the Avizo where you do the deep learning In Dragonfly there is a cost function and you can choose multiple cost functions, but what happens in the learning process, is it presents the network with all of your training data, but not the validation data. It uses that to adjust the weights of the network, then it takes that that network that has learned something, and it processes all of your training data and it evaluates the cost function. Or let’s say the loss function. Then it evaluates it on the validation data, that was not used for updating the weights of the network, it’s never seen that data, it was not used to teach the network. So now you have the loss function on the actual training data, and the loss function on independent validation data Then what happens is the network does

another iteration and shows it all of the training data again. And that iteration is called an epoch. And so epoch after epoch, you can see the network is learning because its loss function is falling. That is, you know, how many times did it pick a seven when it was actually a three? Yow far away was its prediction from the ground truth segmentation? You can see it getting better and better. And so you can set a cut-off, say I want the network to stop training whenever the cost function is below some threshold, or I should say loss function, one of the loss function is falling below some threshold. Or you can just let it train for a certain number of iterations. So it’s very easy to train for multiple multiple cycles. It looks like there’s another question, can we take the processed data from the deep learning algorithm? Oh, we already saw that question. How can you tell it’s not over fitting? Well Benjamin’s question on overfitting, that’s specifically why you want to set aside validation data. If it were overfitting, it would only learn to become more accurate on the training images it sees that it uses to update the network weights. But because you’re evaluating the loss function on independent data that’s not used during the training cycle, you know that it’s not overfitting, otherwise its loss function would never fall on the validation data. And I’m trying to see the chat window, it looks like Nabil just contributed a chat, but I don’t have that open. Question for Mike, we know Python is open source and easy to interpret. How opaque or transparent is TensorFlow? Do we really understand what the software is doing? Yes and no. Yes, it’s completely open source and if you want to learn the mathematics of artificial neural networks, you can download the source code and hack TensorFlow all you want. No it’s not transparent because you do need to be a mathematician or at least mathematically inclined. The average microscopist is not someone who is likely to have the time to go and develop the expertise to become – uh oh, Nabil became (laughing), I thought you were spending your time becoming a better microscopist? I don’t know how you have of enough time to also become a machine learning/ deep learning expert. But the short answer is yes, it’s open. In the end, these networks, they don’t have two weights like the fitting of a line to a function, they don’t have the thirteen thousand weights like the fitting of a neural networks weights to figure out is it a zero, one, or two, or a nine, etc. These neural networks like a simple unit, it has on the order, in the in our case anyway, it is on the order of 22 million coefficients. So you can have networks that are trained on exactly the same data or exactly different data and have similar productive predictive modeling capabilities, they can they can see results, and they may have completely different sets of networks. Just like if I train a five-year-old to read and I trained another five-year-old to read the same textbook, their neural networks are going to be wired differently in their head. But it is open if you want to learn about how TensorFlow works there, are probably about ten million hours of video on YouTube, on how TensorFlow works, and there are dozens and dozens and dozens of papers just on semantic image segmentation with TensorFlow Let’s see other any more questions? Ah, what is the main difference between the Avizo… well let the Avizo guy take the Avizo question. Are there any other questions right now Samantha? Nope, I think that is it. Well for tomorrow we will get into the segmentation wizard and we’ll see a little bit more of the mechanics of how you might label some data and make some predictions, and do it without preparing too much training data. That’s sort of the emphasis is to streamline your workflow, so that, I mean maybe I should what I should say right now is, biologists have been doing segmentation like this for over 20 years, on the order of 30 years, where a graduate student in order to annotate a cell may spend two/ three weeks in front of the computer labeling every pixel. With deep learning instead of labeling every pixel of every slice, maybe you label three slices and then the other 2,000 slices get labeled for free. That is the deep learning algorithm does it for you. But even labeling those three slices can be painful and expensive, and so tomorrow’s lesson on the segmentation trainer is a way of streamlining that, so you don’t spend hours and hours or a day and a half preparing three slices of training data You can spend more a few minutes of your time and then have the computer do the data augmentation, and help you prepare more training data. So that will be the topic of tomorrow with the Dragonfly segmentation wizard