[Ian, Jeff, Sander, Georgia]

Summary:

The lab flooded on the night of Feb 6th around 9 pm. There was about 1-2 inches of water in the main GQuEST lab under the optics tables as well as a bunch of fine sediment that came into the lab. We stayed with the facilities staff and custodial staff to clean it up.

Timeline:

Around 9:50 Jeff came into my office on the second floor of bridge and asked told me the roof of east bridge was leaking. I went with Georgia (Rana's visiting Grad student) and Jeff to check it out. There was water coming from the fourth floor of bridge and going down the stairs all the way to the basement. There was only a little water on the floor in the basement outside our lab so we thought that there wouldn't be water in our lab. We decided to check and found there was about half of an inch of water in the lab (at around 10:15) (First_look_Lab.jpg,First_look_hallway.jpg). Jeff had called Caltech security about the water leak, so we were able to flag down the Caltech security person and facilities staff that had responded to the original flood. 

At this point, the water was filling the center of the lab under the optics tables. The center of the room is the lowest of water and sediment was congregating there and the water had started to move out of the main GQuEST lab and into EE shop. At this point, we started to turn off all electronics that could be affected by water that were low. The first thing we did was turn off the laser so that people could enter the room without wearing goggles. The next thing we did was start to unplug all electronics carefully. As water was in the room, electronics were a safety hazard and needed to be turned off carefully. There were a number of power strips on the ground in the water. We were able to turn them off safely and then remove them from the water. We removed all electrical things from the ground, which includes all cables. We turned off all the power to the optics tables, including power from above. We also did this in the EE shop, although there was almost nothing on the ground. We moved the tool chest away from the water.

At this point, talk to the facilities staff who called in for more backup and immediately started to work to find where the water was coming from. As the water was rising, the bottom of the least of cabinets was starting to get close to the water the cabinet on the south wall of the room contains lots of important optics on the bottom drawer. I took the super optics out from the bottom drawer and moved them to the clean room in the RbQ lab. At this point, Georgia, Jeff and myself grabbed the shop vac from the mech shop and the pump room (Lab_with_shopvac.jpeg). Hoping to be able to vacuum the water up faster than it was coming into the lab. We use the vacuums to suck up water and sediment until the vacuums were filled and then we dumped the vacuums down the shower drains in the bathroom closest to the lab after about 20 minutes of this we found that we couldn't remove water faster than it was coming in as the water was still rising so we temporarily halted. We continued to remove furniture and things that were on the ground and place them either in the RB lab or in the hallway. We removed most furniture that was not tables, server racks, or cabinets. At this point, the facilities staff member had found the source of the water. The door that was sealed off during the construction of the lab is lower than the ground around it and there's a drain that drains water that would collect there (Door_with_clogged_drain.jpeg). For some reason, the drain had been blocked off most likely because of the construction of the new building next to us so water had pooled there and was flowing into the lab. The facilities member got a pump and started pumping out the water (Outside_pumping.jpg). At this point Alan Rice showed up and we started to talk about how we could remove the water from the lab. Alan called in more facilities people and custodial staff. 

At this point, the water had risen to its highest level in the center of the lab. There were probably 1.5 inches of water, which then extended into the EE shop all the way down, partly into the wet lab from the ease shop. It went down the hall into the pump room and was draining down the drain. There was an active flow of water from the door where the water was coming in all the way to the drain in the pump room. Along with the water was a thin layer of sediment every place that the water was. This was probably when the water was at its highest. At this point, because the water was being pumped out much less water was coming into the lab and we started using our shop vacs to clear out the water in the EE shop and pump room. After some time, we started using the shop vac to slowly drain the water out of the hallway and EE shop. 

At this point (around 2 am), custodial staff showed up with industrial vacuums that were able to handle much more water. The vacuums were operated around the clean room to suck up water, but never went into the enclosure except with the head of their vacuum (Lab_Custodial_Vacuuming.png). Within a few minutes, we asked them not to go in with the heads of their vacuums. We worked with mops to pull sediment out from under the tables in the clean room enclosure for the vacuums to suck up. At this point, facilities had outdated the pool of water outside with a few pumps, and it was almost gone (Outside_pumped.jpg). Because no water was flowing in and we were vacuuming out the water, we were able to quickly remove water from the GQuEST lab. After most of the water was removed, the vacuums moved on to the EE shop, hallway, pump room, and wet lab. While they vacuumed out the other rooms, the custodial staff and I used mops to clean up as much sediment as we could from the area (Lab_mopping.jpg). Once as much of the sediment that could be easily removed was removed, facility staff and custodial staff had to move to the basement where a mechanical room had flooded, posing more of a threat to the building. 

We also found that there was a leak coming from the ceiling above the sink in the wet lab. We think that this is a result of the original flooding from the roof of the building that found its way down through the walls and pipes to the wet lab. At the time, we put a large trashcan under it, but it would be helpful to eventually find out where this came from.

At this point, there was still a thin layer of sediment all over the floor, and since there was no water to hold it down, we were worried that any movement of the air could transport that dust onto optics on the tables. The only place this would really be of concern is in the clean room enclosure. Since HEPA filters keep positive air pressure in the cleaner enclosure. The only thing that can really get onto the optics is dust that's in the air inside of the clean room enclosure already. In order to prevent the thin layer of sediment that was on the floor inside of the enclosure from getting blown up onto the optics, I wiped down the floor of the enclosure bit by bit with paper towels. I started by using isopropyl alcohol on the tiles, but it felt like it was pulling up some sort of coating on the tiles, so I switched to water. I cleaned the floor of the enclosure in this manner, using only water as a solvent. The only part of the enclosure I missed was at the center of the optics tables and under the floor rack, which holds the amplifier. I obviously could not clean under where the optical table legs make contact with the floor because I have no way of lifting the tables. To try to clean under there I used water to try and flush and sediment out of there. I am not sure of how successful that was and how much sediment is left under the legs. Because of the cleaning, the floor of the enclosure should be relatively clean, and booties should be worn on the inside of it. After this was done, my main concern became drying out the parts of the lab that have been touched by water outside of the clean environment. I borrowed some of the HEPA filter blowers from the hallways around the building that are owned by PMA and placed them in the EE shop and the hallway outside of the pump room. The worry is that water has damaged the drywall, and to prevent mold, I wanted to dry it out.

The facilities people were able to clear out the drain (Outside_cleared.jpg) and said they would fix the drain and install infrastructure to prevent this in the future. I also included a timelapse from inside the lab that shows the water coming in. It only takes a photo once an hour so the time it is flooded is very short. More photos can be found in the flood folder on the nextcloud.

I then stayed at various facilities, and Caltech people showed up to assess the damage.

Aftermath:

There is still sediment on the ground around the clean room enclosure. We will need to clean around the enclosure to get settlements off the ground. There is also a layer of what I assume is scum from the water that has accumulated on the tile. I'm actually not sure if this clear film that has accumulated on the tile is from the water or if it is actually from the tile and the water has caused it to become delaminated. This is something we will have to consider when we are looking at solves to clean the floor with because I'm not sure if the floor is ESD rated the floor of the clean room is relatively clean because I cleaned it with paper towels, but it still needs to undergo a thorough cleaning the floors of the EE shop, the hallway outside the pump room, the pump room and the wet lab all needs to be cleaned thoroughly in the same manner. We also should probably lift up cabinets and shelves to get all of the sediment out from under them.

Facilities will also probably want to come in and patch up the door that was leaking to access this door. They will probably have to rip a hole in the drywall and that will create lots of dust. In addition to fixing the door anytime drywall is exposed to water. It should generally be replaced and so as part of the lab fixing, they will probably want to replace the bottom 16 inches of drywall anywhere that the water was close to this will be most of the GQuEST lab as well as the pump room. 

Affected Equipment:

This is a list of the equipment that is affected by the flood. It is by no means an exhaustive list.

• There were a number of power strips on the ground that were actively submerged in water. All of these power strips should be replaced. All of the things that were plugged into these power strips should be checked as it is possible that the power strip shorted their power supplies. This includes the flipper mirrors and the Furman
• The Furman power conditioner was very close to the water as it is on the bottom of the rack and plugged into a strip that was in the water. It should very likely be replaced since it operates such sensitive electronics like the seeder and laser amplifier.
• The super-optics were in the bottom drawer of the lista cabinet on the south wall of the lab where there was one inch of water. I removed them and put them in the clean room enclosure in the RbQ lab. 
• There is a Siglent oscilloscope that was in the water and then knocked off of the table. It is probably the most likely damaged piece in the lab. It has been labeled with "potential water damage" and has the serial number SDSMMFCQ6R8108
• In the same drawer as the super optics were a number of photodetectors and fiber equipment. This means that they were not in direct contact with water, but they could be subject to high humidity, which could affect their operation. 
  • 2 Thorlabs PDA50B2 
  • 2 Thorlabs PDA20CS2
  • 1 Thorlabs PDA10CS2
  • Various Fiber equipment included in the attached image
• A Covesion OC3 Temperature Controller was on the upper shelf of one of the floor racks. It was about 16 inches above the ground and got slightly splashed with water. Serial number COV1444
• The BNC cables that were on the ground in a bag. This bag got wet, but hopefully, the cables inside are okay.
• The amplifiers for the OFC EOMs are on the bottom of the server rack. The Amplifiers were never in the water, but their power supplies and the BNC cables that feed them were in the water. We may want to replace the amplifiers' power supplies and check the actual amplifiers.
• Our shop vac was used to vacuum up lots of the water. It should be able to handle the water easily, but we did not install the water filter for it in the heat of the moment, so it may have sediment inside it.