Medical Image Segmentation F.A.Q.

Written By T. S. Tasi

One can find online tens of thousands of journal articles, technical reports, theses, dissertations and summarizing analyses on segmentation and anatomical model creation based on medical images. Skimming through hundreds of pre-selected items that apply to a particular problem set, or reading several comparisons of algorithms can be quite time consuming. It can be especially frustrating when the expectations of the project timeline indicate preliminary results / milestones that are due relatively soon. We’ve all been there, or will get there, eventually.

This Q&A list attempts to tackle frequently brought up topics that were picked from several forums, groups and sites related to the subject in the title, mixed in with typical smaller issues that I have seen come up on multiple occasions.

Note #1: In its current state, this post contains absolutely no code snippet for any, potentially ‘coding-level’ problem. Depending on time constraints this might change in the future.

Note #2: Therefore, it many not offer the reader an immediate option to present results by running some piece of code on the currently available data. It may, however, very well save hours / days / weeks worth of research time instead. So in case you actually prefer to think about the problem before you start hammering those keys on the claviature, you're welcome to proceed.

Note #3: For decision-themed question items there is no added analysis of decision-making authority regarding the topic at hand. It is assumed that some form of control - or an influence on the decision at least - does exist.

Note #4: I’ll reserve the right to extend the current content as I see fit, and/or fix typos, grammatical errors, etc. as needed.

List of the F.A.Q. items:

  1. If I can choose between different image sets with various formats, e.g. jpegs, tiffs, pngs, dcms, npzs, etc. which one should I pick?

  2. I might be able to get access to a machine, but the whole acquisition process looks complicated. I think I have a good enough set of images to start with, anyway. Should I even ask anyone to co-sign this?

  3. How can I even validate the results of the segmentation process? Who decides what is “good enough”?

  4. Image acquisition has been scheduled. Is there anything I can do to ensure a good outcome data-wise?

  5. Should I use [ insert latest and greatest algorithm family here ] for my problem?

  6. I have just heard of this [ insert newest machine learning approach here ] method recently, should I spin off an instance to see what happens?

  7. I want to design the entire software architecture ahead, is that recommended?

  8. What is the likely accuracy of my results if I do everything ‘by the book’? Is there a way to improve?

  9. I have a choice between CT images or MRI, which one should I base my method on?

  10. Is it possible, based on the literature, to simply just find a good range of HU values that correspond to my region of interest?

  11. I want to create a digital 3D model of the selected anatomy, what’s the easiest way for me to achieve this?

  12. How can I perform operations on my three-dimensional models, or make them 3D printable?

  13. I think I have a missing slice in a specific sequence. How can I confirm this? What if the data is non-recoverable?

  14. I have slice-per-slice preliminary results, and a subset of them contain open, C-shaped, unclosed objects. How can I fix this?

  15. (A relatively loosely related question, nonetheless, decided to append it.)
    I have a task to perform post-operational assessment in 3-D. Is there a recommended, preferably well documented, approach to this?

 

Q1: If I can choose between different image sets with various formats, e.g. jpegs, tiffs, pngs, dcms, npzs, etc. which one should I pick?

Almost always chose the DICOM set, if available - get familiar with the tags and metadata it provides, as soon as humanly possible.

Alternatively, select the most standardized one out of the available sets. Make sure to record the acceptable ranges of the image properties, tighten the restrictions as hard as you can. (Needless to say the same applies for DICOM sets.) Once the rules are set, strictly check and re-check the set of images when appending to it, immediately throwing away any item or sample that does not meet the set of criteria. Basically make a serious effort to avoid circumventing the rules already in place. Possible image encoding and compression issues plague methods that rely on, e.g. jpg images, especially, in many cases resulting in an algorithm that works as intended exclusively on a single medical dataset.

 

Q2: I might be able to get access to a machine, but the whole acquisition process looks complicated. I think I have a good enough set of images to start with, anyway. Should I even ask anyone to co-sign this?

If adequate equipment is at hand, and there is a possibility to create an image set, make a conscious effort to do it, and do it in a standardized fashion. Always chose the ‘real data experience’ over the artificial set or the 3rd party acquired approach. Your supervisor’s approval will be needed for this, of course, but just in case you’re so in luck that you can work with a CT, MRI or even a conventional X-ray machine, make a push for it. Search for the ‘scan protocol’ of a similar product on the market or the anatomical region you’re doing work on. Do some market research on the top products in the field, and study their approaches, it will pay off.

After doing some preparation - don’t take too much time though - talk to a radiographer, technician or radiologist, if there’s one on the team. Try to find one if there isn’t, and analyze the related scan protocols or review the latest images together. If there is a slight chance to talk to people who work at any of the PACS workstations, and you can make it happen, take the chance! In case you happen to be working on your bachelors’ or masters’ or similar, keep in mind that interdisciplinary topics and procedures for such research are available at most institutions, seek them out! Sure, obstacles may present themselves along the way, but most radiologists will be happy to help you if you can help them in return with some task that could be easily automated with a little effort. It is assumed you have some coding skills and image processing knowledge at this point.

ct_acquisition.jpg

CT acquisition of some artificial bones for fitted guide testing purposes. Scan protocols for such objects are evidently significantly different than the ones for human patients with dense bone structures, soft tissue, etc. Still, from the viewpoint of understanding the process and creating the medical image segmentation pipeline, it provides a lot of useful, preliminary lessons to all involved.

 

Q3: How can I even validate the results of the segmentation process? Who decides what is “good enough”?

This can be a bit challenging initially, at least. As for development of software as a medical device, the answer should be obvious: in-place internal processes and work instructions compliant with regulations of appropriate regulatory bodies, a defined set of unit-, integral- and system level verification tests, a pre-defined set of validation tests of the device, so on and so forth. Often, these do not exist yet for the particular application, since development just started on the project, or perhaps the set of tests are far from complete, etc. In such event, software developers need to introduce some tools to help confirm that they are indeed moving ahead in the right direction.

There are two useful auxiliary modules - which, in many cases, the radiologists involved in the development process may require to be implemented anyway - that can assist in this matter. I would recommend implementing both, assuming the timeline allows for it:

  • 2D DICOM slice browsers with contouring / mask overlay

    • This is the classic approach that has many demo implementations across different platforms that handle DICOM medical data.

    • An intensity selection tool that displays the corresponding Hounsfield value at the position of the cursor should be added to this, similar to a ‘color picker’ tool for standard images, to confirm the validity of the boundaries.

    • Slice number indicators should be included as well to make sure the slice position is recorded at all times.

    • Depending on the segmentation approach, either a mask to highlight the segmented region or the contour of the boundary line shall be displayed, whichever is more intuitive for the particular task. These need to be clearly visible so that they really stand out, for the user to identify foreground and background segments easily.

    • Another benefit of developing the listed functionalities is that it forces the registration process to be accurately accounted for. Any error in the any of the registration steps should reveal itself almost in an immediately identifiable manner.

  • 3D visualization of the 2D-3D registration of the original slice sets and the resulting models

    • Applicable only if the end result segmented models are expected to be observed in three-dimensional space specifically.

    • The demo app referenced below could be a good starting point, for instance. The example only implements an intensity thresholding solution of marching cubes to generate a surface model, one can read more about the method on its wiki page and here as well.

    • The challenging part, of course, is replacing the latter method with one’s own specialized segmentation algorithm, while maintaining the correspondence of the coordinate systems.

    • The 2D planes that show the corresponding image slices should be scrollable / draggable, in a user friendly fashion.

    • A function to toggle the display and opacity of the segmented object(s) should be added, as well.

    • Similarly, the benefit of developing the functionalities listed here is that it will force the registration process to be accurately accounted for. Again, any errors in the steps responsible for the registration should reveal themselves almost immediately.

A radiologist, or a similar team member with comparable medical knowledge, can use any of these - or potentially other tools with an equal set of provided functionality - to check the results. Their goal, specifically, is to rule out the possibility of over- or under segmentation occurring in critical areas of the patient’s anatomy, to avoid working with incorrect models that do not represent the underlying anatomical features well. Expect them to be highly critical of such areas, and that in general they will demand more insight and more tests to be added.

TestMedicalDemo3.png

The demo application ‘Medical Demo 3’ of the Visualization Toolkit at its core should be a great starting point for a module that is used for assessing the result of the segmentation process.

Link to the specific VTK Examples page on the Kitware GitHub site:
https://kitware.github.io/vtk-examples/site/Python/Medical/MedicalDemo3/

 

Q4: Image acquisition has been scheduled. Is there anything I can do to ensure a good outcome data-wise?

Yes, absolutely there are several key action items to be aware of. Depending on the size and maturity of the organization, of course, there might be processes, work instructions and procedures in place for this, and the collection of these will streamline the process a lot. However, there might not be that many guiding principles in use, which means each individual involved has to think ahead out of self-diligence in most cases. This could be turned into a novel opportunity to introduce a whole new process as well, exploring the options available, and making your own demands regarding the acquisition process. This is the occasion to be as greedy as possible to avoid that extra workload in the long run!
Some of the most important items that come to mind regarding medical image acquisition are the following.

  1. Be aware of the scheduling, identify key players in the process.
    Develop a relationship with these individuals, in case you are allowed to have a hands-on approach defined by your role. Even if the latter is not really the case, being on stand-by, listening to these key actors will be extremely useful in understanding what is actually going on behind the scenes.

  2. Have the scan protocol(s) ready at hand at all times, both in digital and print-out formats.
    Explain what is needed, what the parameters are. Make them read the documents, which should be short and concise, and have them pay attention, in a preferably non-invasive manner. This could be the hardest to achieve, but not an impossible task to accomplish with time, nonetheless.

  3. Demand couch removal to be activated on the workstation software.
    In case you are working with a relatively modern CT or a magnetic resonance imaging machine, this should be a no brainer. It is expected to be a simple toggle on the interface, and it should work almost 100% of the time, depending on the quality of the machine, of course. This can also save you a significant amount of time when performing preparational clean-up steps on the image processing side of things. If it happens to be not available - or it is claimed to be non available on the machine - ask for the accompanying software manual. Again, developing a relationship with key people could prove to be beneficial, so that they will try to assist you in your quest rather than trying to block your requests.

  4. Have metal artifact reduction (MAR) activated as well, if available.
    For this item, certain software modules and possibly even hardware upgrades might be necessary, which is likely out of your reach. However, modern machines have this option embedded in the system by default, needing only a license upgrade in many, since the feature itself is usually locked by design. Search the software manual if necessary, explore what subsystems and features are available. In the event that you find yourself in need of removing metal artifacts on the images, this could very well make a night and day difference in the quality of the segmentation results.

metal_artifacts.png

Metal-related artifacts, depending on the implant or metal component size, can have severe effects on the imaging outcome, especially in computed tomographic images. More often than not, patients already have some sort of implant in place near the region of interest.

Applications like iMAR by Siemens Healthineers, or O-MAR by Philips can help immensely, check out their corresponding pages for demos and examples.

 

Q5: Should I use [ insert latest and greatest algorithm family here ] for my problem?

In general, absolutely not.

A good rule of thumb is to 1) start with the most basic methods, see what types of algorithms could be applicable, and 2) try to narrow down the search space as much as possible. Have a specialist - who has the knowledge to carry out segmentation and verify future results, if needed - perform a slice-by-slice contouring on a set of image samples, so that there would be a baseline available. Unfortunately some explanation might be necessary to make sure that everyone involved understands why this is important, including project and/or program management, since this could take significant time to perform properly.

By this time the scan protocols should be taking shape as well, simultaneously evolving with the implementation. Regarding the protocols, try to restrict the placement of the region of interest (ROI) that contain the target ‘objects’ in the images to the greatest degree possible, so that the eventual method can work in the narrowest, best defined area. This could be done relative to a well placed and/or easily recognizable mark or a collection of markings, or even carried out on a coordinate system level. To achieve this, most likely several brainstorming sessions will be required and info on the set of tools available need to be common knowledge that is shared between the participants.

Once contouring / labelling, whichever terminology makes more sense for the case, is considered done on an initial set of images that represent the potential inputs well enough, usually a series of measurements take place. The following items in the checklist below might be worth examining, regarding the target area, or possibly the entire target volume and its surroundings. It is assumed that the preferred end result is some form of volume, defined by a sequence of well demarcated areas, each on separate, two-dimensional image slices. If not, disregard the below items only relevant for volumetric results, in case there is no way of rephrasing them to make any sense for that particular problem.

  • What potentially distinguishes the target (usually called foreground) from its surrounding area (background)? Are there any commonalities? Can it be primarily based on:

    • intensity values,

    • distribution of these intensities,

    • partitioned specific areas that have similar characteristics (divide-and-conquer),

    • a well-defined shape, that

      • potentially can be ‘described’ by a formula later on,

      • or a generic shape that can be customized for the patient,

    • textural characteristics,

    • etc., and any combination of these and others not even mentioned.

  • For results that make sense from a volumetric point of view, look for any overarching pattern between the attained results on each slice, there might be something to exploit there.

    • Density histograms based on slice positions can help define the parameters of the expected slice-by-slice results, based on their position in the scanning space, for example. The positional component is easily extracted from the strict protocol-based scan results.

    • The linked article describes building a probabilistic atlas based on this information, assuming a large enough dataset has been acquired or is accessible.

  • Check if there is a way to gradually narrow down the target area or volume by a set of steps, that each utilize one of the previously mentioned techniques. Each, not overly complicated step that gets the team closer and closer to the desired target is usually worth the effort invested.

If there is one thing to take away here, it is that working solutions are generally not based on a single “bold stroke”, e.g. by applying some active contour model on the raw images, and calling it a day well spent. More often than not, there are no shortcuts to be made.

Trying to find the perfect parameters for that single, ‘all-embracing’ algorithm, that will provide a solution for every scenario, is usually a tremendous waste of time.

 

Q6: I have just heard of this [ insert newest machine learning approach here ] method recently, should I spin off an instance to see what happens?

If you are fortunate enough to have access to, either

  • a pre-trained model that you can just simply "plug in" to do some of the job, and build around it, or

  • a well annotated dataset, labeled by professionals with a sample size of tens of thousands, and the time, the budget and the know-how to build from the ground up

then for all means go for it! Since both of these are rather uncommon, in almost all other cases the answer would have to be a firm no. Unsupervised learning and reinforcement learning algorithms are conveniently left out from this analysis. Both have started to attract the attention of researchers only recently, due to the fact that by design they do not seem to necessarily need all that high-quality annotated data, although the required sample size is usually still an issue. Real world applications of such methods, to my knowledge as of current date, are yet to be seen.

To get an accurate picture of where artificial intelligence-based approaches can step in and solve some of the problems in medical applications, more specifically regarding image processing and medical image segmentation, i. a., this open access journal article is truly recommended. Going through the data a few recognizable patterns can be quickly discovered.

  1. Most of the approved software as a medical devices (SaMDs) fall in the Radiology category: an area of specialty that benefited from the usage of the DICOM standard, in my humble opinion, the most.

  2. After digging a little more it becomes apparent that these devices:

    • work with highly standardized imaging with strict scan protocols, and

    • solve well-defined, highly focused smaller tasks with ML within their workflow.

The U.S. Food & Drug Administration (FDA) not long ago published an AI/ML SaMD Action Plan based on a previously published proposed regulatory framework for AI/ML-based SaMDs, of which the latter was intended to be a ‘Discussion Paper and Request for Feedback’. While the former document might be temporarily unavailable at times, the latter is still accessible. As it is a good discussion-provoking 20 page report, for those that are interested in the topic and plan to work on ML-based medical software in the future, I would recommend spending the necessary time on reading it thoroughly.

 

Q7: I want to design the entire software architecture ahead, is that recommended?

No, not really. In the prototyping phase the goal is pretty unambiguous: create a valid PoC (Proof of Concept) first, that is actually viable and possesses practical application potential. There is no need to hinder the creation process by setting up any serious architectural design requirements right from the beginning, and take the focus off the construction of a general, working prototype. Surely some very basic aspects can be considered, e.g. not making integration into a vertical prototype - that demonstrates the components and functionality of a specific core feature - impossible, but this rarely occurs in reality.

poc_brainstorming.png

One of the earliest sketches of a DICOM browser that eventually led to the construction of the basis of the desktop segmentation module. The design shown here was quickly revised afterwards, and only attempted to describe the inner workings of the proof of concept, rather than establishing how the components of the system should interact with each other. The purpose of such drawings is definitely not to ‘box in’ the module in any way regarding implementation architecture, but to start a conversation going between software developers working on the project.

In terms of system design, once the technology stack is shaping up and the algorithmic approach is clear, there are only a couple, closely related key characteristics to consider here. The expected level of automation and the possible user expertise needed for assisting the algorithm to perform its task. In fact, I think it is clear that these two are just the opposite sides of the same issue. The number of the subtasks that can be solved entirely and reliably by the software itself - proven by the usual procedures - is somewhat inversely proportional to the breadth of the professional expertise and the volume of training material that is required for its potential users. I’m not specifically referring to the expertise and trainings mandated by regulation, but the effective knowledge and critical attention needed to operate that precise module. Meaning, the more the machine can sort out by itself, the less critical workload the human has to carry out, eventually.

Finding a good balance is the real aim here. Over-automation is not a feasible solution either, e.g. manual selection of a few control points on a surface is usually way more quick and efficient than trying to find complex topological features.

The chosen segmentation algorithm could be a decisive differentiating factor, affecting every element of the system, all the way up to product-level user stories. The user’s required knowledge base and the supplementary tools required to be implemented will most likely highly influence the entire experience, application workflow and, eventually, the hiring demands for new positions at the organization, as well.

 

Q8: What is the likely accuracy of my results if I do everything ‘by the book’? Is there a way to improve?

Simply put, there’s no way to improve after reaching a certain threshold. The errors made during each step will eventually accumulate in the end. Some might think this a good problem to have, but it really is not. It will somewhat feel like hitting a brick wall.

There are a few articles published that address this issue, one of them I would like to highlight is a clinical paper on computerized treatment planning of cranio-maxillofacial surgeries. It details the measurements made in each step of the process, from the acquisition of the images, all the way to the simplification and post-processing of the surface models for - possibly - getting ready to be manufactured. Even for dental applications that deal with scans of such dense structures with associated high intensity values in CT imaging, it is remarkable that every image processing step in the pipeline impacts the accuracy of the resulting digital models in question.

The paper tackles the accuracy issue of oral surgical planning with the eventual end goal of fitting dental surfaces, so the process does require a lot of sub-steps, that would not pose a risk individually. Nevertheless, the presented results are still eye-opening and noteworthy. The resulting models are impacted at all “stops” along the way, e.g. image acquisition (resolution), binary segmentation, 2D smoothing, surface generation, smoothing of the surface models, etc.

One can imagine how the effects of similar image processing steps could be worse for the accuracy of models of more difficult, e.g. soft tissue structures that do not possess high intensity characteristics. Especially, if accompanied by a complex segmentation algorithm working in the background, which is likely more error-prone than a relatively clear-cut thresholding. Unfortunately, for medical applications designed to be used in practice, well documented “real world” testing of their accuracy seems unavoidable.

 

Q9: I have a choice between CT images or MRI, which one should I base my method on?

Short answer:
Start with CT images and accompanying methods, and over time “graduate” to using MRI if, and only if, it is decided that it shall be necessary for higher accuracy, or for providing an alternate option for users.

Long answer:
The decision is usually up to the specialist doctors involved, but there are certain scenarios that make using magnetic resonance imaging (MRI) essentially inevitable:

  • high level detail required in neighboring soft tissue, organs, ligaments of the region of interest (ROI);

  • the entire segmentation ROI consist of soft tissue, ligaments, or organs;

  • patient has too many - still MRI compatible - metal-based implants, that would make the CT images heavily obstructed by artifacts.

Both are capable of producing what the radiologists refer to as ‘3D image data’ or just ‘3D images’. The term is intended to classify image data that capture the entire 3-dimensional volume with proper slice thickness and slice spacing that permits that type of acquisition, not leaving any gaps between any pair of slices.

One major advantage of using CT data is far easier standardization and simpler creation of widely accepted scan protocols. For MRI scans it can be far more problematic to construct a general scan protocol with so many different settings and practices available already, e.g. T1, T2-weighted, various fat saturation techniques, etc., with a wide variety of machines and implementations of these standards. There is basically little to no guarantee that the same settings applied on different machines of diverse generations and manufacturers would provide comparable image sets.

Medical device companies are often forced to differentiate their scan protocols based on the scanning machine’s manufacturer: either by placing them into separate documents or inserting huge tables of necessary settings in their protocol descriptions.

 

Q10: Is it possible, based on the literature, to simply just find a good range of HU values that correspond to my region of interest?

This kind of question usually emerges right after one flipped through an opening slide show on a radiology-related medical image processing workshop, or just completed a similarly titled course. These introductory lectures typically contain one of the typical, simplified charts of the Hounsfield scale, with intervals titled ‘air’, ‘fat’, ‘water’, ‘soft tissue’, ‘bone’, etc. Some variations do try to add more sub-intervals, e.g. distinguishing between trabecular bone and cortical bone, to introduce some granularity in the overall picture. As a general rule it is advised to regard these values as a mere ‘suggestion’ only.

If one considers bone tissue alone, numerous examples can be provided that overthrow the simplified theoretical intervals of the previously mentioned charts. Case in point, the images below present such instances. The first one shows the actual min/max bounds of the HU values of the circled area of the anterior femur surface of [-9, 192], and the mean value of 65.53, as captioned in the image. Better yet, the other one shows an ellipse-bounded area of a maxillary bone with the mean value of -460.48, and standard deviation of 262.45. (Obviously, different software tools were used for the two cases, since the type of data displayed is somewhat different.) These are edge cases of course, however, it is not uncommon that relatively simple methods of segmentation fail at these crucial areas, further complicating the necessary algorithms or simply rendering them useless all together. Overall, it is considered very rare that any attempt that utilizes the intensity values only would actually provide a general working solution.

femur_anterior_surface.png

Anterior surface of the distal femur, just below the patella.

  • Min / Max: -9 / +192

  • Mean / Standard Deviation: +65.53 / +54.62

  • Surface Area: 0.08 cm^2

maxillary_bone.png

Maxillary bone, below the eye socket.

  • Centroid Position: +5.44, +49.13, +106.40

  • Surface Area: 29.07 mm^2

  • Surface Perimeter: 4.99 cm

  • Mean / Standard Deviation: -460.48 / +262.45

 

Q11: I want to create a digital 3D model of the selected anatomy, what’s the easiest way for me to achieve this?

In order to determine the easiest and the most efficient way to create a 3-dimensional model, the purpose of the model needs to be considered. What is the exact goal of the model, what will be achieved by its creation?

Is the 3D model aimed at (please interpret this list below as a gradually growing set of requirements, a new subset added at each list item):

  • A) Exclusively for providing a 3D view of the ROI (and occasionally performing simple visualization functions)?

    • Volumetric, voxel-opacity based visualization can be performed in a relatively easy way, compared to a mesh-based solution.

  • A) Providing a 3D view and also B) facilitating selection of control points, definition of auxiliary line segments, or planes, etc.?

    • Any widely accepted and utilized method based on polygonal meshes should be appropriate in this case. No additional requirements - other than demanding that the structure should be clearly visible, with no obstructions - should be introduced in this circumstance. Although it has to be mentioned that clear visibility often implies the existence, and careful application of certain mesh operations (for instance, removing smaller connected components under a certain volume threshold) that perform the cleanup necessary to get the right results.

  • A) Providing a 3D view, B) facilitating the selection and definition of particular geometric structures, and C) producing 3D objects that supervised geometric operations can be performed on?

    • Depends on the required set of operations, of course, but this is evidently the most difficult scenario. The range of operations could include non-complex global translations and rotations, local alignments with existing third-party models, or even rather complex boolean logic on a set of polygon meshes - often necessary for producing compatible objects that can be 3D-printed.

    • The suggested avenue in this case is to take a slice-by-slice approach. Have strict requirements for each resulting 2D cross-section produced by the algorithm for the corresponding slice, and make sure that all conditions are satisfied before accepting that specific result.

    • In addition, interpolation techniques could be used for inter-slice approximations, or even for filtering out anomalies. This specific implementation with contours as an ITK remote module demonstrates the idea. It’s not the exact details that are important here, but the concept behind the implementation is what counts, the solution itself could be based on, e.g. binary shapes as well.

    • Often a separate step for manual inspection and potential correction is embedded in the segmentation ‘pipeline’ itself. It is there to guarantee that the slice results are all correct, before an attempt is even made at creating the 3D model. Frequently, PhD students, post-docs and/or technicians carry out this demanding task for each patient case.

    • As the 2D cross-sectional preliminary results are finalized - assuming a binary outcome for each pixel in the image here (1 / 0 for foreground / background), thresholding is always an option if the assumption is wrong - the final volume can be constructed, at last. The discrete version of the marching cubes, which is a variant of the classic method that works on labelled datasets, comes in handy for this task. For instance, the Visualization Toolkit provides such a method in vtkDiscreteMarchingCubes, although many other implementations are available online, in several image processing packages and frameworks.

    • Provided that the slice thickness is adequately small, obtaining smooth surface models might not be an issue. Nevertheless, since the marching cubes method

      • tends to be sensitive to stray voxels and small imperfections, and

      • tends to produce ‘spiky’ surfaces from time to time,

      it could be essential to perform smoothing on the resulting model. The key is not overdoing it at this step.

    • As soon as the models are ready, it is common practice to separate them based on mesh connectivity to extract individual models for each anatomical structure. Specific functions can be written, or existing ones can be utilized for this purpose. The individual models then can be saved / uploaded to the desired location. Common formats include STL (“Standard Triangle Language” or “Standard Tessellation Language”) or PLY (“Polygon File Format”), and thankfully they are documented well enough that modifications are possible, if needed.

 

Q12: How can I perform operations on my three-dimensional models, or make them 3D printable?

3D printers generally expect single layered, closed, watertight meshes as an input - for a nice formal definition please take a look at this referred article. From the question itself, we can safely assume that “operations” refer to boolean operations (typically union, intersection, and difference) on polygonal (normally triangular) meshes for this topic, since these, at least in theory, do not alter the necessary ‘printability’ properties of these objects. Tackling these, one could:

  • create their own functions that perform these operations from scratch (likely very time consuming, and not recommended thanks to the enormous amount of edge cases and exceptions),

  • alternatively build upon existing libraries (perhaps the best choice), or even

  • solely use the available, existing functionality in one of the available packages (personally rarely seen this work, since there are several missing or problematic methods in virtually any of the libraries that are accessible on the web).

Fixing non-manifold vertices and edges, holes, disjoint surfaces, self-intersecting faces (or non-manifold / non-Euclidean geometry, in general) after the three-dimensional models have been created is commonly viewed as an approach to avoid. (A strict tool for detecting such areas of concern could be MeshLab). Quite a few reasons can be listed, namely:

  1. the endless possibilities of different configurations that one could encounter,

  2. the sheer quantity of these problematic configs that could be present on the models that were created without any precautions,

  3. the manual nature of fixing each that can not be automated, or

  4. in the worst case, any attempt of an automatic fix on certain surfaces could cause an explosion of non-manifold modeling errors on the surface area, which is unfortunately also a possibility.

Instead, only a small, finite set of additional requirements need to be added to certain steps described in the answer to Q13:

  • for the 2-dimensional cross section results make sure all the closed objects are filled, and only objects intended for segmentation have any associated pixels present, and also

  • for the sequence of preliminary results confirm that only the intended objects are connected both in the image plane and in the results for adjacent slices (vertical direction), as well.

Re-meshing should also be considered as a preferred operation in between steps, and auxiliary functions should be implemented to verify each geometry. For mesh resampling / re-meshing, I would encourage the reader to check out the existing methods on the subject, such as the ACVD project.

The next image pair below show some of the noticeable properties and differences between meshes that correspond to an out-of-the-box surface mesh generation algorithm (on the left), and surface meshes that were created by a set of tools that were implemented with the aforementioned points in mind (on the right).

Triangular meshes generated by the classic marching cubes method (on the left) and a custom made segmentation process (on the right) of a CT scan of a knee joint. The marked / highlighted areas are explained as follows:A - Some unrelated objects, formed by polygons that can be associated with the scanning table (sheet) in this case, are still present in the image on the left, but not in the image on the right. It should be noted, however, that these differences can be overcome easily with simple filters and settings.B - Anatomical parts - just the patella, in the present case - that are not required for the planned operation are still present on the left, but not on the right.C - Visible holes and unconnected components can be seen on the left side, especially in the circled areas. These can contain hundreds, thousands of polygons that would need to be re-examined or supplemented to make the structure watertight to ensure 3D-printability.D - Each bone segment is a single layered, closed, watertight connected component, each eligible for 3D printing.E - Careful segmentation of adjacent slices make it possible to separate components that should not be connected, so each bone segment can be displayed, processed and stored individually.

Triangular meshes generated by the classic marching cubes method (on the left) and a custom made segmentation process (on the right) of a CT scan of a knee joint.
The marked / highlighted areas are explained as follows:

  • A - Some unrelated objects, formed by polygons that can be associated with the scanning table (sheet) in this case, are still present in the image on the left, but not in the image on the right. It should be noted, however, that these differences can be overcome easily with simple filters and settings.

  • B - Anatomical parts - just the patella, in the present case - that are not required for the planned operation are still present on the left, but not on the right.

  • C - Visible holes and unconnected components can be seen on the left side, especially in the circled areas. These can contain hundreds, thousands of polygons that would need to be re-examined or supplemented to make the structure watertight to ensure 3D-printability.

  • D - Each bone segment is a single layered, closed, watertight connected component, each eligible for 3D printing.

  • E - Careful segmentation of adjacent slices make it possible to separate components that should not be connected, so each bone segment can be displayed, processed and stored individually.

 

Q13: I think I have a missing slice in a specific sequence. How can I confirm this? What if the data is non-recoverable?

Let us take a little detour here. By the time the process reaches the segmentation stage, the imaging data is usually either received as a packaged archive, or via an external media. Depending on the hospital protocol, the imaging department might use third party tools that could scramble the slices in different folders - even between different series(!) - and furthermore, could give them randomized filenames to make sure each series cannot be reassembled or opened without the provided viewer application, that is also attached to the media or packaged with the archive. The viewer application, of course, often limits the data that can be retrieved or accessed from the image series. From what I observed this annoyance is a common practice in relatively smaller hospitals or institutions. In this case, separation of the individual series into individual folders need to happen. It is highly recommended to automate this process (we will not go into implementation details, since this is out of the scope of this question), however, it can be done manually, with a tool that is usually some form of a general DICOM viewer / editor application. The Sante DICOM Viewer used to have this functionality, the Lite version unfortunately does not, however there are dozens of these tools available online, if needed.

From this point on, we assume that each series is in its own directory. As a recommended next step, it could be useful to check that the Slice Thickness (0018, 0050) values, the Spacing Between Slices (0018, 0088) values, image dimensions, acquisition (study) dates, etc. are each uniform for the whole sequence, just to make sure everything is aligned. Naturally, which tags we examine here could also depend on the previously defined scan protocols, too. One could utilize the DICOM Standard Browser for exploring relevant metadata, in case navigating the DICOM standard becomes confusing after a while. Once these preliminary checks are performed, we can proceed.

For correctly identifying the missing slices it is not advised to rely on the slice number (0020, 0013) at all, since it is not a mandatory field. Our best bet is checking the series of values provided by the Image Position (Patient) (0020, 0032) tag extracted for each slice of the image sequence. If there is an unexpected jump in the ordered series of z coordinates that does not align with the previously examined slice thickness and spacing observed, it is proof that an item is missing from the sequence. Ordinarily this “jump” can be identified visually, as well, just by browsing the image sequence. Nonetheless, if presenting a proof is required, the previously described method provides this in an unmistakable way.

At this point this insufficient data event needs to be reported to the appropriate persons since the entire series is required to perform high quality segmentation. If caught early, it is often possible to simply retrieve a proper image series from the raw data stored in the corresponding PACS storage, if present. Note that this ‘raw’ image data is usually lost after a few days, since it does take up large disk space. It might have already been overwritten by the time it is recognized that the particular dataset is inadequate. In case the data is truly non-recoverable, scheduling another imaging appointment for the patient will have to take place. This is highly inconvenient, interrupts the usual chain of events and typically causes a major delay in the whole process.

It should be emphasized that data of sufficient quality and quantity is a non-negotiable requirement for medical image segmentation.

In the rare event that recalling the patient for imaging purposes is not possible - due to the severity of the situation, most likely - and the segmentation has to be performed by a close deadline, a quick solution would be to use some form of interpolation to obtain the segmentation result that correspond to the missing slice in the dataset. This option shall be exercised only in cases of emergency and if, and only if the dataset allows it, by:

  1. having a single missing item in the sequence, and by

  2. having a particularly small slice thickness for the entire series (< 1 mm), e.g. 0.625 mm is considered sufficiently small.

Even then, I personally would strongly advise against using this technique, since it can lead to over- or under-segmentation, especially if the missing slice represents a critical section in the area that the procedure is performed in.

 

Q14: I have slice-per-slice preliminary results, and a subset of them contain open, C-shaped, unclosed objects. How can I fix this?

From what I can see, this particular question appears on numerous forums, blog posts and message boards related to image processing and image segmentation, specifically. In the resulting discussion, responders and commenters usually enter into arguments on why morphological closing is the only appropriate answer, regardless of the used platform of course, attempting to discredit any other potential solution suggested. While closing, in general, works well, there are scenarios in which it is not directly applicable. For such an example, consider something similar to the objects presented in the image below. Previously not connected components, that should stay unconnected, might become connected as a result, changing the topology, or simply producing an unusable result as a consequence of a large structuring element, and so on. Closing typically performs well in cases where the opening is relatively small, so the ratio of the object size versus the structuring element size needed is still relatively high.

binary_image_slice.png

A binary image, that is a result of a “half-baked” segmentation process, which desperately needs some clean-up.

After the tiny components are deleted, and the area corresponding to the couch is wiped entirely (lower left corner), the largest connected foreground component can be separated. The remaining individual object, which is a binary cross-section of a patella in this case, should be able to be corrected by performing the steps mentioned.

Another solution would be to:

  • perform aggressive cleaning outside of the true ROI in the image,

  • isolate the object (connected component) in question on a separate layer, turning any other foreground objects to be part of the background, so that it would not interact directly with other (foreground) objects in the image for the time being,

  • perform the Euclidean distance transform (by using, for example, scipy.ndimage.distance_transform_edt) on the inverted image,

  • find the local peaks (e.g. using skimage.feature.peak_local_max) in the distance image with the labels of the original binary, mark the first (largest) label of the peak array (e.g. by utilizing scipy.ndimage.label),

  • perform a watershed (e.g., skimage.morphology.watershed, or the new skimage.segmentation.watershed, etc.) on the negative distance map with the marker array, and with the inverted image being the mask,

  • and finally assign every pixel that does not correspond to the largest region (which should be the background at this point) to be part of the foreground.

It might be necessary to tweak these steps to be applicable to the specific relationships between the objects in the image(s) in question, to actually make it work, however, it should not take too much effort. At least some combination of these suggested steps should be adequate to close the objects one-by-one, after which they can be “merged back” into the original. Needless to say, results acquired this way should be checked and re-checked during a manual inspection step in the segmentation process.

Let us remind ourselves of the required visualization capabilities of the software that uses any similar methods. The developers have to make sure that there is an available option which makes it possible for the end user to view the entire scanned area / volume: just in case having a “look around” near the region of interest is necessary.

 

+1: (A relatively loosely related question, nonetheless, decided to append it.)
I have a task to perform post-operational assessment in 3-D. Is there a recommended, preferably well documented, approach to this?

The closest I could find to the approach that has been used is the AURORA protocol. The linked article explains the concept clearly, with accompanying figures that describe the process. I should note, however, that using 3-D imaging for pre-operational planning, as well as post-operational assessment is usually hard to justify, regarding the dose that the specific patient would receive during multiple CAT scans. Based on the calculations and the subsequent recommendations of the radiologists, there is a high probability that this approach will be approved only in special cases where it is warranted. One has to be prepared for performing accurate determination of post-operational positioning where the post-op 3D CT scan data is simply not available, which evidently involves various, mostly 2D-3D registration steps, as well. Refer to this whitepaper, for instance, to get information on what the latter approach could look like.

Referenced resources:

T. S. Tasi
Previous

Enhancing System Resilience - Elemental aspects of hardening Ubuntu
Next

Techniques: Mesh Modeling