Working for manufacturer engineering couple years. Sharing the manufacturing experience, skill, DFx, design, and information. Be care of this blog is personal share and content may not be 100% correctly.
This article shares a real case from Workingbear’s past experience dealing with an urgent customer complaint. One day, Workingbear was suddenly assigned to fly to a factory in China to “hunt for oil.” My manager even warned, “If you don’t find the source of the oil, don’t bother coming back!” So, with a heavy heart, Workingbear set off—didn’t even dare to book a return ticket.
The reason for this emergency was a serious DOA (Dead On Arrival) complaint. The product completely failed right out of the carton box when it reached the customer.
The defect? The physical keypad didn’t respond when pressed. The key travel (stroke) felt normal, and there were no signs of stuck keys. However, pressing the buttons failed to activate the circuit. Interestingly, most of the unresponsive keys were in the 3, 6, and 9 positions—just like a telephone keypad layout. At that point, the failure rate was about 1.12% (35 out of 3000 units). Since the complaint came from one of the company’s biggest customers, everyone was on edge.
The frontline customer service team quickly reported the issue to the technical analysis team. When they opened up the product, they were shocked to find a layer of white powder and oily residue on the circuit board beneath the keypad. It was obvious the contamination happened at the factory since the tamper-evident seals were still intact.
What puzzled everyone was that all products were supposed to undergo 100% final functional testing before leaving the factory. After that, they also went through another 100% quality check at the distribution center. So how did this DOA issue still make it into the customer’s hands?
Another strange point: the keypads used metal domes, and on the production line, operators were required to clean the keypad contact areas (gold fingers) with IPA (isopropyl alcohol). To avoid trapping any IPA that hadn’t fully evaporated, the factory even spaced out the cleaning and enclosure assembly steps by inserting two operators in between. So where did the white powder and oil-like residue come from? No one had a clue.
To get to the bottom of the problem, the company formed a special task force. It included people from customer service, quality assurance, R&D, and outsourced factory management. Since manufacturing was outsourced, the meetings also involved the contract manufacturer. From then on, it was daily meetings, progress tracking, and non-stop updates. That was the most exhausting part—having to report new findings and update the status every single day.
Short-Term Countermeasures
The first thing the company did was to take immediate short-term actions to stop the issue from spreading. They re-tested 100% of the units from the same batch for functionality. Then, they selected six of the passing units and put them through high and low temperature cycling to simulate aging, just to confirm that the re-tested units wouldn’t fail again.
The next step was to ask the factory to provide all serial numbers for units produced in the same series—including the ones made before and after the problematic batch. Using those serial numbers, the team pulled matching units from the shipping center’s warehouse for inspection. Unfortunately, similar defects were found in other units within the same series. This further confirmed that the issue originated in the factory’s production process.
Root Cause Investigation
To find the real root cause, the task force created a fishbone diagram (also known as a cause-and-effect or Ishikawa diagram) listing all possible sources of the contamination. They carefully checked every suspect area. Even though the factory insisted that the oil and white powder didn’t come from their equipment or process, we pushed them to provide a complete list of all possible sources of contamination.
We investigated every step—SMT PCB loading, pick and placement, reflow, hand soldering, ICT (In-Circuit Test), FVT (Function Verification Test), repair, final assembly, and even the anti-static air guns. Each one was thoroughly examined.
This failure analysis gave Workingbear a whole new perspective. For solderability-related issues, EDX is useful because it reveals the elemental composition—like identifying the IMC layer or phosphorus-rich layers. But if you’re trying to identify chemical compounds or surface contamination, you need FTIR, which compares molecular structures, unlike EDX which only identifies elements.
From the FTIR results, part of the contamination appeared to come from a cleaning agent—either IPA or glycol ether. Another part seemed to match the chemical signature of flux.
Cleaning agent residue made sense, since we already knew the factory used IPA to clean the gold finger traces on the PCBs. But seeing a large amount of flux residue was strange. Under normal circumstances, that shouldn’t happen.
The most reasonable explanation? An operator mistakenly used flux as if it were IPA and wiped it directly on the PCB. We simulated this scenario and confirmed that if you apply flux to the gold fingers and let it sit, a white powdery layer eventually forms. It also causes the keypad to malfunction. However, in the actual defective units, the contamination wasn’t quite as severe as in our simulation. That made us suspect the residue might be a mix of IPA and flux. More testing would be needed to confirm this, and we had to check whether it was even possible for operators in the factory to confuse flux with IPA.
So far, the investigation was still ongoing. But since we had found some solid clues and a clear direction, Workingbear finally felt a bit of hope—maybe he’d be allowed to fly home after all!
Leave a Reply