Investigating Analytical Tools for Testing Cleanliness with Foresite's Eric Camden
At the West Penn SMTA Expo, I met Eric Camden, lead investigator at Foresite Inc. He was quite busy most of the day but towards the end, we found time for a chat. In the short time we talked, he taught me a great deal about fluxes and analytical tools for testing cleanliness.
Patty Goldman: Eric, Tell me a little bit about Foresite and yourself.
Eric Camden: Foresite is an analytical third-party testing lab that focuses on the process and all the materials that go into assembly. We do failure analysis as well as process qualification, process audits and optimization. We cover everything from incoming raw components to the final packaged product as it leaves the door. We have a long history of dealing with OEMs and CMs and looking at both the failure side of the process as well as the process qualification side. What we've learned in our 26 years is that everything matters. It's a cumulative process when it comes to building an assembly. We look at everything from the innerlayer of the PC fab all the way to the ESD bag that a board is put in on the way out the door.
Everything that comes in contact with an assembly has an opportunity to cause an issue. Today I was here specifically speaking about no-clean flux processes—how to properly wash them, and what happens if you don't—and I also gave a presentation on localized cleaning. We kind of ran the gamut in terms of different people that I've talked to and different topics, but there seems to be a very large market out there cleaning no-clean flux and all the associated risk and the right parameters to use to remove all those fluxes.
Goldman: You are largely a testing laboratory, correct? You don't have any cleaning product or anything like that, do you?
Camden: Correct. We don't have any cleaning product. We do some recovery cleaning for some of our customers if they find something has been in the field that wasn't properly processed the first time around and they can't get it back through their process in time. They'll ship it to us, we'll clean it and ship it back out. But primarily we're a process materials lab so we do cross sections, SEM/EDX, FTIR, XRF, X-ray, ion chromatography, IC mass spec, we have a lot of equipment that helps us determine the quality of a product both before it goes out the door and after it comes back from the field in a failure mode.
Goldman: Are your customers generally the assembly companies or the OEMs or is it a mix?
Camden: It is a mix. Typically, the CMs aren't a direct customer. They're being driven by their customers to use our services. But there are certainly many larger CMs that you can think of that have been our direct customers for many, many years now because they've seen what happens if they don't do certain types of analyses before a product goes out and how it tends to come back as a failure. Over time it's a lot easier to do the work up front than it is to do it on the back end because then you've got a failure. It’s not only the money it takes to replace those but the potential of losing business going forward due to issues that the assembly caused their customer by failing. It's a lot easier and cheaper to do all that homework up front to determine if cleanliness is an issue or the material is an issue with the product in the field. If you do the homework up front and you determine that it either is or is not proper for the application, we can help the customer determine that before building 100,000 of these a month and getting them in the field.
Goldman: So, reliability is the name of the game.
Camden: Reliability is definitely the name of the game. And it's really something that we promote in terms of what our services really render in the end. It is reliability, because that's what everybody is shooting for in this industry. No matter if you've got a 30-day warranty or a 20-year warranty, you have to do the right analysis up front to determine what your level of reliability is going to be. You can't always put a date on it and say, “Okay, it passed these three tests, so it's guaranteed to last 20 years in the field,” because of a lot of unknown variables in the end-use environment that play a role, but that's very hard to calculate during upfront testing.
But we can say in general that someone is using the right material, using the right process, the right equipment and parameters. In the end it says okay this board appears to be exactly what it needs to be in terms of reliability because you've got low levels of ionics. Your PC fab looks like a good piece of material. Your components are clean as they come in the door. So those kinds of things, and again because what we do here in the electronics industry is such an additive process you've got suppliers sending in 20 different types of materials that you're putting on one card. Once you marry all these things together now your resultant cleanliness is coming from all those different suppliers, not just what you're doing.
Goldman: It's complicated.
Camden: It can be very complicated.
Goldman: And I take it you may perform an audit function in part, right?
Camden: That is correct. It's really a function of what we do every day just by looking at someone's process or someone's product because we're looking at how their equipment and how their parameters handle that set of materials. In one sense that is an audit, but an onsite process audit is also something that Foresite offers. I'll get on a plane, go halfway around the world and put my head under your machine because I want to look at your equipment, and your operators, and I want to see how they're doing things by hand. Humans are one of the worst things possible in terms of reliability. Because you can't calibrate one next to the other and repeatability is key in this business at fixing problems. It really goes a long way into optimizing your process. It can be considered an audit in a couple different senses of that word.
Goldman: And you're also a troubleshooter.
Camden: Very much so. Because if you bring me a product that has failed, we're going to use our techniques to determine what the root cause of that was and then we'll take that information and we'll go back to the assembly house and say, "Okay, it was part of this process that caused the failure so let's see…was it your parameters themselves, was it the equipment not functioning as you thought it was?" And we help better determine how to optimize those processes. We troubleshoot the process based on the analytical findings and then we're able to optimize that to where that failure doesn't occur anymore.
Goldman: That's good. What is the history of your company?
Camden: Terry Munson is our president and founder. He founded the company 26 years ago and I've been there for 18 years as of January. I've done a lot in 18 years. It's remarkable that, after 18 years, we're still dealing with a lot of the same basic problems when it comes to manufacturing. And that's being able to properly process a no-clean flux or completely remove a flux residue or understand how these things interplay with one another in terms of reliability at the end. After 3,000 some customers and doing this for 26 years you tend to see most of it. I would never say we've seen it all because every now and then we are surprised, but being in the position that we are, we see so many different failure modes.
It really helps us with our new customers going forward. They say, "We've got this problem we've never seen this before." Well, we've dealt with that 50 times now. We're able to kind of fast track some of these things. We're not just poking in the dark like you must do sometimes on an unknown failure mode. We're able to take our experience and our knowledge and really laser focus in on what should be the issue. And obviously it's not going to be 100%, but most of the time we find that what we've seen with other customers we're able to fine tune that analysis and focus down to one or two different things.
Goldman: Of course, in 26 years technology has advanced so that even though you see the same problems they're on much finer features and much bigger components and all the problems with that. But it still helps you focus in on them.
Camden: Absolutely. Because you know there's only so many ways that you can solder a board. You still must have a component, whether it’s through-hole or surface mount, you’ve got to have solder, and you’ve got to have flux. In general, all these things have held true since day one. You must have these components and then there are a lot of ways to do that correctly and there are even more ways to do it incorrectly that will reduce that reliability. Whether we're working on an old PGA plated-through-hole or if we're working on a brand new QFN pattern that nobody's ever seen, we know that those same things that fail the older through-hole type components, those same issues will plague these finer pitched SMT low standoff type components. Like the QFN—especially the QFN we've seen so many problems with. When you see enough of these failures you end up fighting the same fires, just a different forest.
Goldman: Or you could say different customers and different configurations.
Camden: Right. Because they're all using solder paste, and they're printing it. Every line is basically the same. We've seen a lot of ways to do it wrong.
Goldman: Probably more wrong ways than right I would think. There's always more.
Camden: Yes, for sure. There's only a handful of ways to do it right.
Goldman: Getting back a little bit to about yourself. You've been 18 years with Foresite but before that were you in the industry?
Camden: I was not. I kind of came in blind as a bench tech 18 years ago. I got an entry level job and after about a year or so of just learning what these acronyms meant, I was doing ion chromatography. It was a whole different game 18 years ago than what we're doing today. We had two very small systems and it was all manual. But I worked around some very smart people who are still in the industry. And I just took an interest. I took notice of what they were doing. They were putting these boards under these microscopes all day every day and I was trying to figure out why. What are you looking for and how did it get there? I was around some good smart people who wanted to teach me, and I really took an interest to it.
Then I started looking at some of the IPC standards and how that applied to what we were doing. They’d say, does this pass? And I had no idea what a pass was versus a reject versus a process indicator. Once I started following up more and more on the visual side of things, that led to looking at the chemistry side of things and how we were testing with ion chromatography and how that related to failures or not. It really was just a natural interest for me because there are a lot of different things to branch out into. When you see a failure, it can come from 100 different things, and our job is to figure out exactly what that failure came from, what that material it came from.
At Foresite, we've always had a good library of chemistry signatures for flux types or PC fabs, things like that. It's doing something different almost every day when I'm looking at different customers but it's all under the same umbrella. So that helps because I know if it's flux related I know it's going to come from one of four different sources. Whether it's paste or spray flux, hand soldered or whatever it might be, it's going to be under that same umbrella. It was always very interesting to me to learn more and more about the assembly process. Looking at LEGOs, you need this piece, this piece, this piece for this whole thing to come together. But if you're not quite right it's going to fail, and I really liked the idea of digging into somebody else's process and determining how to optimize that to bring them back to making a good reliable product.
Goldman: Is there ever a time you've come across something that you just couldn't figure out?
Camden: Well, there are some head scratchers. Like I said earlier, we have a general idea when we see a failure mode; we start by looking at topics A, B, or C, in that order. And then sometimes if you get down to J, K, L that's where it gets to be kind of frustrating and when you've got to start looking outside the box. We've tested everything that we know to test. Then you really must start using your knowledge of the assembly process, going all the way back to prefab manufacturing, to figure out where to find this failure mode.
And sometimes—we started doing this 12−13 years ago—we really started focusing on bare fab innerlayer cleanliness that's causing issues. The miniaturization of boards, blind vias, buried vias, things like that. If there's ragged drill we found you can shorten the space between an innerlayer ground plane and a via, and now if you don't have a good solid insulative space between that via and that ground you can have innerlayer leakage that you never see on the surface of the board. You may or may not be able to detect it even with the best of X-ray. We started looking at cleanliness of innerlayers because we saw some leakage event. And it's not always CAF, the historical CAF, but there are other ways to get that kind of innerlayer shorting that we hadn't seen before then. We try to adapt when we've run the gamut and come out with nothing at the end of it. Then we see that it was one of those issues that we saw where we started focusing on the innerlayer bare board cleanliness.
Goldman: You keep going back further and further in the process.
Camden: Yes, until you get back to a handful of sand. Was the sand contaminated that we used for this wafer that was used for this chip? Sometimes you have to go all the way back to the beginning.
Goldman: And there's always something to learn.
Camden: There seems to always be something to learn, especially in this industry because there's always somebody trying to push the envelope. Embedded devices is where we're headed next—you get handed something that looks like a bare card that has all the resistors and capacitors built into the board itself. I think there's only so many places we can go from here. When you think back to printed wiring boards, when everything was hand wired and wrapped, to looking at your cellphone today. That would have filled 10 server rooms for the same kind of power just 30 years ago. We've made leaps and bounds, and when you start talking about aerospace and implantable medical, these devices must be pristine. And we really enjoy being on the forefront of some of these technological advances where some of these customers are coming to us and saying that they want to make sure that they’re putting this out there right the first time, because it is mission critical. Sometimes it is literally life or death with some of the stuff we work on. We must be sure that what we're giving is really good information. And so far, obviously we're still open, so it's a great place to be.
Goldman: I would venture to say your business is only going to increase by leaps and bounds as we go into 5G and beyond. We think electronic devices have exploded now, but I think there's going to be a much bigger explosion, shall we say, of electronics everywhere. And some of that like automotive is going to need to be super reliable. There is going to be a whole lot more that you're going to inspect.
Camden: Right. Then you start thinking about networks, not just wireless charging but over the air kind of charging for your cellphone, things like that. You hear those hints and rumors out there that your phone will always be charged because it's just in the air. There are things out there that we're very much looking forward to and hopefully we'll play some part in that in providing reliability data.
Goldman: I'm thinking more than ever you're going to be called upon for reliability issues or reassurance or something similar.
Camden: Absolutely. Really that's what it comes down to. Working where I work, doing the work that we're doing, we really are a reliability lab. Because whether we look at your product upfront or after a failure analysis, we're either instilling the reliability upfront before the initial product release, or we're looking at your failures and determining how to optimize the process to build the reliability back into that process before it goes back out the door.
Goldman: Of course, you always hope that you get that upfront opportunity to prevent.
Camden: Sure. And like I said earlier it's a lot cheaper to do it up front.
Goldman: What else can you tell me about your talk this morning?
Camden: It was on two very important issues. Apparently, it was a good turnout here for rainy Wednesday morning in West Penn. There were quite a few people in the audience with a lot of good questions. People that are working with no-clean fluxes and they're cleaning them at the direction of either their customer or some of their internal engineers think that it needs to be cleaned. One of the main problems with cleaning a no-clean flux is what you leave behind. No-clean flux based on the amount of rosin or resin in your material can be extremely difficult to clean. Especially under QFNs, micro-BGAs, the low standoff components, because you've created this barrier that's very hard to penetrate. QFNs are complex under the best conditions and even when you've got a solid ground plane you can't get the gasses to vent out properly and you certainly will struggle to clean those properly.
Any time you've got a no-clean flux, that outer shell is meant to remain in place and become basically a poor man's conformal coating. If you partially clean a no-clean flux, what you're doing is you're ripping off the band-aid. You're exposing the wound. And if you leave any of that material behind that was meant to be bound within that outer resin shell, now it's very hydroscopic, it's very conductive. It can be as bad as water-soluble flux residues.
In the first talk this morning we discussed some of the failure modes of no-clean. It all comes out to electrochemical migration, electrochemical leakage. These issues are very prevalent in the industry for people who are trying to clean no-clean. And just because you can't see a residue doesn't mean it's not there. We touched on making sure you have the right equipment parameters and making sure you're using things like belt speed, temperature, impingement angles, and saponifier concentration. A lot of these little things play together in concert to make an effective wash process. And then being able to do localized extractions and ion chromatography after you've cleaned, to improve the effectiveness.
Historically, the ROSE tester has been used, but that is going by the wayside for several reasons, and rightfully so in 2018. It was never meant to be used as acceptance criteria. It was supposed to be process monitoring to see if something was completely out of whack somewhere in your process. You would catch that. And the materials that were around when the ROSE tester was implemented are no longer being used at all. These materials that that tester was meant to test have gone by the wayside, but the tester was still in place and the criteria had never changed. But that is coming to an end to a certain degree. And a lot of engineers are really seeing the benefit of not using the ROSE tester for anything more than just the occasional process monitoring tool.
When you start looking at a process is, you really need to focus on parts like QFNs because they're the hardest to process whether using no-clean or water-soluble flux. If you focus on parts like that and you end up doing localized extractions, obviously I'm going to promote the C3 because it's an automated localized cleanliness tester, but it does a 0.1 square inch area extraction so you're able to look at very specific areas of the board. And what you don't do versus ROSE or even standard extractions for ion chromatography is you don't normalize out these pockets of contamination. For instance, if you've got a rework area and it's not very clean when you're done and then you take that whole board and you try to do the cleanliness measurement by dropping that in five gallons of IPA deionized water, you've got one square inch area that is dirty and you've got 148 that are clean. Then when you put in your dilution factors for anything like that you're taking length time, width time, two times 10% for population factor. You've normalized that out over the entire surface of the board.
I've seen more and more over the last five, six, seven years where people are understanding that full board cleanliness doesn't give you the information that you need. You need to look at these hard to process areas and components and you need to look at what that data says. You do your localized extractions and there's many ways to do it, some are more effective than others. Then you take that solution and you do ion chromatography on it. Ion chromatography we believe is the be-all-end-all when it comes to cleanliness testing. It tells you exactly what the residues are in both type and amount—and where they are when you're looking at localized extractions. And that's why it's so important to look locally and not globally when it comes to cleanliness.
Your higher standoff, larger-pitch type components are much easier to clean so you focus on a QFN on a micro-BGA, any of the bottom terminator components like that that are very difficult to process. If you focus on those, if you get those clean, set up your wash parameters and your thermal profile right to process those, then you'll be in good shape for the rest of the board. In general, we've found that to be true over the years.
So that was the first talk. And then the second presentation I gave today was on localized cleaning—hand cleaning after hand solder or rework or whatever that operation might be. Just like any other wash process there's a lot of right ways and a lot of wrong ways to do it, and we've seen both. To give you a synopsis on that, basically knowing where your effluent is going, being able to control the amount of solvent you're putting down to do the cleaning and then knowing where that spreads to. Once you bring a solvent into play with a flux residue you've made that a mobile phase containment basically. If you've got neighboring components that have a low standoff, well that surface tension of that solvent and flux residue is so low it's going to easily wick its way underneath SMT type components or whatever it might be. And if you don't do a good job of rinsing that whole area then you're just doing contamination relocation. I've heard it called vacuuming without a bag. The area that you're cleaning looks good, but it's just spreading it everywhere else.
Like any other cleaning process, it's verifying that it is effective using the right analytical techniques. There are a lot of wrong ways to do localized cleaning, but if you know to test not only the area that you cleaned but create a cleanliness map around there, looking at all the neighboring components, you can say, "Okay, I've cleaned this area of interest but I'm also not contaminating the neighboring components."
It's been a good day here at the West Penn show because a lot of people have come up to me since the morning presentations and we've had further conversations on what their companies are doing.
Goldman: That's great. The big thing about the no-clean versus others is that they're notoriously hard to clean. And I'm just curious about this. Why don't people then go back to fluxes that aren't no-clean?
Camden: This is one of my bugaboos about no-clean. I believe the people who process their boards with no-clean flux and then wash them, they are in the mindset that if they partially clean a no-clean flux it doesn't matter because it's a no-clean flux and what's left behind will be non-conductive and non-corrosive. But that simply isn't the case. If I see somebody that's going to do a wash process no matter what, don't make it harder on yourself. Use the water-soluble fluxes.
Goldman: So if you use a flux that needs to be cleaned, aren't you further ahead then?
Camden: There's zero doubt in my mind when I look at a water-soluble board that's been washed that if I see residues, they are conductive. If you're processing with no-clean and you don't fully remove all the residues and you see some white haze here and there, that's mostly likely flux residue left behind. You can't look at that and say, "Yes that's active. Yes, that's corrosive. Yes, that's conductive, it's going to cause a problem." Given the option, process with the water-soluble flux and not a no-clean if you're intention is to wash. I think really at this point like with a lot of changes in the industry it's just education and time. We need to teach these people who are requiring the washing of a no-clean flux that what you're doing may be more detrimental than leaving it in place.
Goldman: Just get the regular flux out there.
Camden: Or leave the no-clean on there. Visual inspection is one of the worst criteria for determining a board's quality and in turn its reliability. I've been doing this for 18 years and I can't look at a board, I can't look at a residue, and say that's good or bad. There's just no way to tell by looking at a board. But there are so many companies out there that say, "I want a clean, shiny solder joint. And that's how I know it's a good quality board." Well, think about when they went to lead-free, they didn't have shiny solder joints anymore. What you thought was the be all end all criteria has now gone away because you were forced to go lead-free. I think it will take some time the same way with the cleaning of no-cleans.
Goldman: Yeah, everybody thought no-cleans meant there was nothing there. That's not true.
Camden: Correct. It's lower risk. There are lower active components to it to begin with, but that doesn't mean that you have no risk of contamination or no residues. It's just a matter of making sure that it's been thermally processed to a level that whatever is left is near benign. But that's just not always the case.
Goldman: Well, this has been interesting. And I thank you for your time.
Camden: Perfect. Thanks Patty.
