Accomplishments IceCube Project Monthly Report April 2009 Interviews for seasonal driller positions were completed and positions filled. Support and logistical requirements for the 2009 2010 Season were submitted via the SIP process. During April 2009, the overall detector uptime was 95.6% and the clean in-ice uptime was 85.7%. Most of the lost clean time was dedicated to further DAQ improvements. The data warehouse enhancements were completed in April. The new improved Production Data and Production Simulation file systems are working well. Detector Commissioning and Verification as well as Online Filters and Triggers, are ready for IceCube 59 string (IC59) data runs. In preparation for moving to the IC59 run the Amanda TWR data system was shut down on April 1st. We are now running in IceCube only mode. As of mid-april we have resolved insufficient data throughput problems and are close to issuing a new DAQ release. Final IC59 test data is being verified by the physics working groups with a plan to transition to IC59 in mid May. North-West side of the Amundsen-Scott South Pole station covered with its winter snow coat, and the Milky-Way in the background. Picture: courtesy of Patrick Cullis
Cost and Schedule Performance The project is 90.1% complete versus the plan of 89.9% complete, as measured using earned value techniques. Remaining contingency is $7.0M and the contingency as a percentage of remaining work is 26.4%. The contingency continues to be carefully managed to ensure the successful completion of the MREFC project. Page 2
The schedule variance at the end of March 2009 is a favorable $385K (last month was $350K). The variance is related to RPSC s schedule performance during the FY08/09 Pole Season. The earned value cost variance at the end of March 2009 is a favorable $1,758K (was $1,559K last month). RPSC positive cost variance of $992K (last month was $913K), is due to lower than planned labor and On-Ice support charges in FY2008 and in FY2009. The $498K favorable cost variance in Implementation (last month was $475K) is the result of lower training costs and a labor ramp down ahead of schedule. Contingency Status and Plans during the month of March 2009, the total Remaining Contingency has been decreased from $7.3M to $7.0M after implementing Change Request IC0142. Total Contingency % of the cost-to-go has been slightly decreased from 26.8% to 26.4%. Description Date Approved Total Baseline Allocated Budget Allocated Budget Change Contingency Budget Budgeted Cost of Work Remaining Contingency % of Remaining Work Status as of December 2008 275,293 267,520 0 7,773 29,589 26.3% CR 0138 Request to procure Iridium Modem, 02/04/09 275,293 267,640 120 7,653 Disc Array and perform a CPU Upgrade CR 0139 Request to procure 3 additional 02/04/09 275,293 267,802 162 7,492 Disc Arrays CR 0140 Add effort to correct Pole data and 02/25/09 275,293 267,966 165 7,327 reprocess taped data CR 0141 Procure a Disc Array for PSU 02/25/09 275,293 267,973 7 7,321 Status as of January 2009 275,293 267,973 0 7,321 28,728 25.5% Status as of February 2009 275,293 267,973 0 7,321 27,356 26.8% IC0142 Add NPX SE effort to 1.9.3.2 04/01/09 275,293 268,271 298 7,022 Status as of March 2009 275,293 268,271 0 7,022 26,650 26.4% During the 4 th quarter of Project Year 7 (January 2009 March 2009), the contingency percentage of remaining work increased slightly from 26.3% to 26.4%. Page 3
Item Risk Assessment & Potential Contingency Adjustments 1. Contingency likely to be required to mitigate technical, cost and schedule risks associated with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining. 2. Pre-Operations for additional capacity enhancements to the data storage and network systems both at the South Pole and in the Northern Hemisphere, and for extending software development efforts. 3. RPSC FY08-FY11 Baseline Re-Plan including the FY08 Closeout (-$600K) and FY10 Fuel cost savings (-$400K). 4. Cost of the limited amount of remaining instrumentation for an 86-string detector, 80 strings from the original proposal plus the six deep core strings financed by European partners. The additional instrumentation is 5 surface-to-dom cables and ~100 DOMs Estimate ($K) $2,097K $1,000K -$750K $1,350K 5. RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. The current RPSC baseline already includes support for 80 strings. 6. Project Year 7 Closeout for all WBS items excluding RPSC and the ANG. The closeout will return authorized budget to IceCube contingency due to lower drilling training and travel costs and staffing ramping down ahead of plan. 7. The cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. This conservative estimate is based on RPSC's Rev 8 Budget. 8. Potential cost to retain experienced key personnel in PY9 to ensure the success of the last drilling and installation season and the timely completion of the construction project. $1,340K -$300K $1,050K $700K 9. Potential additional Fuel Cost exposure during the last season of drilling (FY2011) $450K Available Contingency as of Mar 31, 2009 Total $6,937K $7,022K GPRA FY09 Mid Year Estimate (U.S. budget only) U.S. Budget At Completion (BAC) $237.4M U.S. Planned Value through FY09 (BCWS) $220.0M Planned Value % Complete of the BAC 92.8% Drill Operation and Installation Procurements and repairs to the Enhanced Hot Water Drill Equipment (EHWD) are underway. The outline for an EHWD documentation and end-ofproject turnover package is complete. Work on fabricating a surface hose-handler and people-mover sled is in process. Additional snowmobiles were procured and retrofitted for the South Pole. Parts for a prototype heater (Sioux) were secured and an installation plan was drafted. Preparation of the Test-Bed at the UW Physical Sciences Laboratory for the off-season training is underway. Interviews for seasonal driller positions were completed and positions filled. All support and logistical requirements for the 2009 2010 South Pole Season were finalized and submitted via the SIP process. Page 4
Detector Commissioning and Verification Commissioning and verification tasks are complete and the detector is ready for IceCube-59 (IC59) data taking. During recent test runs, verification efforts played an important role in diagnosing problems such as the EMI signals in some newly-deployed modules due to a meteor radar experiment located near IceCube, and finding dropped DOMs. The team is developing a test that compares a known good average waveform for each DOM to the one obtained in the current run. Preparatory to that the team took the first simple step and added a plot of the average waveform for each DOM to the verification-monitoring web page. Long-term studies of IC40 data will begin once IC40 data taking is completed. An initial good run list for IC40 running was released. Data Acquisition Hardware and Software During March 2009 the DAQ development team worked on the Cactus release for IC59. This release will fully support readout of the IceCube detector in Soft Local Coincidence (SLC) mode both for the in-ice and IceTop sub-detectors, and provides special triggering support for the newly deployed DeepCore array. Several performance problems manifested only when the full 59 strings and trigger set were deployed and tested at South Pole with the result that the data throughput was not sufficient to keep up with the IC59 trigger rate (approx 1900 Hz up from 1250 Hz for IC40). As of mid-april we resolved throughput issues and are close to issuing a release of Cactus in order to start the IC59 run. Several test runs were taken with SLC enabled. The data rate is about a factor of two higher than estimated due to a higher than anticipated rate of isolated hits and also due to a miscalculation during estimation which did not properly account for buffer overheads (the technical explanation is given below). We are working on amelioration strategies because the observed data volume (~20 MB/sec event data volume to disk) is problematic for downstream processors. The main contributing factor to the +100% increase in data rate comes from an improper estimate performed in October 2008 with early test data. At that time a single string was taken out of DAQ and run in SLC mode in order to estimate the 2009 run period data rates. Unfortunately, using a single string was not sufficient to demonstrate the additional event volume due to readout of nearly empty strings: the readout of a single string has a buffer header that is approximately 100 bytes. This was a much smaller fraction of the total readout buffer when the detector was operated in HLC mode, which had larger data volume associated with the non-header information in the readout (the hits) because (a) the hits are themselves larger and (b) there were typically many HLC hits in the string. Now, the presence of isolated hits, many of them noise hits, reduces contributions of both (a) and (b) resulting in a relatively large volume of readout header in the event. Page 5
IceCube Maintenance and Operations Detector Up Time 95.6% IceCube Clean Run Up-Time not including AMANDA Array 85.7% Unscheduled Downtime 0.6% Events from DAQ 3.25 billion The IceCube detector ran and took data without major issues during April 2009. The overall detector uptime was 95.6% and the clean in-ice uptime was 85.7%. Most of the lost clean time was dedicated to further DAQ development and long test runs of the soft local coincidence data collection. As of this writing all known issues with SLC have been solved. Two previously established milestones were completed in time: 1) The end of combined TWR+IceCube operations and the permanent deployment of the new Live experiment control software, and 2) decommissioning of the previous anvil code. Still missing is the transition to full IC59 operation, which is expected to happen in May after the filtered test data taken with IC59 is checked by physics working groups and filter proposers. During an IC59 test run an unusual trigger fluctuation pattern was noticed. Upon more investigation it became clear that some DOM triggering rates were changing over time and that there were unusual waveforms (with sinusoidal oscillations) in a large group of DOMs (approximately 100). These effects were traced within days to the interference from a 10 kw, 46.3 MHz antenna at the South Pole surface (from a Meteor finding experiment called COBRA). To mitigate this issue a DAQ configuration was developed that raised the PMT gains and the absolute value of the discriminator threshold to match. An early version of that configuration was deployed and tested with the antenna in the full power operation mode and also during quiet antenna time. Both runs showed clean behavior. With this configuration in place, the radio antenna is not a roadblock to start IC59 operations. Page 6
Data Handling South Pole systems operations in the IceCube Laboratory (ICL) continued through April with systems operating smoothly. There were occasional stops to the IC40 run for test runs in preparation for IC59. The figure below shows the daily satellite data transfer rates. The data warehouse upgrades were completed in March and April. The migration to the new data/exp and data/sim with improved file system performance are working well and production data processing and production simulation are no longer hampered by limited disk space and performance. Online & Offline Filtering, Software & Database The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IC40 and Amanda TWR data streams continues at South Pole sending filtered data sets over the satellite. In preparation for moving to the IC59 run the Amanda TWR data system was shut down April 1 st. We are now running in IceCube only mode. There were several issues in deploying the IC59 DAQ system that caused some delays in the IC59 run start. These included higher then expected data rates due to adding the isolated SLC hits, and the discovery of some RF pickups on the DOMs from a nearby experiment at the South Pole. Final IC59 test data was collected at the end of April. The test data is being verified by the physics working groups with a plan to transition from IC40 to IC59 in mid May. The major work in online filtering and software systems was: 1. Continued development of IC59 online filter and triggers (completed in April). 2. Continued work for release of new PnF filter system at start of IC59 run. This will be the final major release of the PnF filter system and will include improved performance capable of running with the full IceCube array, improved monitoring and logging with integration into the IceCube Live system, and more robust process distribution framework for the reconstruction workers. Page 7
3. IceTray V3 development work moving forward. This includes work to separate out some core software for sharing with the KM3Net collaboration, which has adopted the IceTray framework for the data analysis system. Offline Level 2 filtering and high level reconstruction of the IC40 data continued to make progress last month. Continued mass production of high-level reconstructions at the IceCube Data Center is going well. Processing of satellite-filtered data for IC40 is going smoothly. Simulation IceSim, IceCube s Monte Carlo simulation program, as such is fairly mature now, and the development has been somewhat limited to improved performance and fine tuning. This task is moving more to the physics groups with the focus of the central project on program maintenance and production running. Education and Outreach IceCube winterovers participated in the 100 Hours of Astronomy's Around the World in 80 Telescopes with a live webcast from the South Pole, on April 4 th. IceCube uploaded text, images, and video to the 100 Hours website. This year marks the 50th anniversary of the signing of the Antarctic Treaty with a celebration in Baltimore, Maryland. IceCube had an exhibit at the Maryland Science Center as part of the events surrounding the Antarctic Treaty Consultative Meeting (ATCM). A team from the University of Delaware attended to talk about Polar Science. On April 4 th, IceCube participated in Science Expeditions, a hands-on open house on the UW campus featuring science research programs and projects. Quality Assurance and Safety The IceCube Quality & Safety Manger will be participating in a design review sponsored by the National Science Foundation and the National Ecological Observatory Network Project (NEON) Project. The monthly reports are posted at IceCube Monthly Reports. Meetings and Events Science Advisory Committee Meeting at UW-Madison May 20 21, 2009 Driller Training at UW-Madison July 25 August 7, 2009 Collaboration Meeting at Humboldt University September 21 25, 2009 Joint Software Meeting with Antares/KM3Net September 25 27, 2009 Page 8