Accomplishments IceCube Project Monthly Report - December 2008 Drilling and string installation at the South Pole is about one week ahead of the schedule. Ten strings were successfully installed by the end of December 2008 and a total of 16 strings were deployed by January 15 th. The deep core prototype string was the 16 th string installed. IceTop installation work for the current season is complete, tanks are filled with water, and the controlled freeze of the water in the tanks is underway. Additional computing hardware was installed in the IceCube Laboratory (ICL) at the South Pole for DAQ, processing and filtering software, and calibration and verification systems. A tape based storage system was installed at the IceCube data center in Madison. IC40-string operations during December 2008 achieved an overall uptime of 97.3%. The IceSim MonteCarlo event simulation program is well advanced and further development is limited to the continuous improvement of the program. An IceCube collaborator was evacuated from South Pole for a non-injury condition and is recovering off-ice. There are no work related injuries this season. Drilling at the location of the 5 th hole of the season, the 45 th overall, and concurrent deployment at the location of the 4 th hole of the season. Photo on December 13, 2008 by Tom Gaisser Cost and Schedule Performance - The project is 88.8% complete versus the plan of 88.7% complete, as measured using earned value techniques. Remaining contingency is $8.6M and the contingency as a percentage of remaining work is 28.9%. The contingency continues to be carefully managed to ensure the successful completion of the MREFC project.
Page 2
The earned value schedule variance at the end of November 2008 is a favorable $197K, slightly reduced from the variance last month. The variance is mostly related to the RPSC s FY2008 schedule performance variance. The earned value cost variance at the end of November 2008 is a favorable $1,662K, an increase from the $1,459K reported at the end of last month. RPSC has a positive cost variance of $657K due to lower than planned labor and On-Ice support charges in FY2008. The $622K favorable variance in Implementation was the result of lower training costs than planned, lagging invoices, and the ramp down of some labor ahead of schedule. Contingency Status and Plans The contingency % of remaining work increased from 27.5% to 28.9% during November. The PY8 (April 1, 2009 and beyond) planning is nearly complete and will result in about a $1M draw on contingency, less than in previous annual planning cycles. 11/30/08 IceCube Total Project Budget Baseline - Change Log ($K) No. NA NA CR0136 NA Description Status as of September 2008 Status as of October 2008 Extend Verification at Clark-Atlanta U. Status as of November 2008 Total Baseline Allocated Budget Allocated Budget Change Contingency Budget Budgeted Cost of Work Remaining Contingency % of Remaining Work 275,293 266,638 0 8,656 33,315 26.0% 275,293 266,638 0 8,656 31,461 27.5% 275,293 266,645 7 8,649 275,293 266,645 0 8,649 29,947 28.9% Page 3
The cost of fuel for the FY2010 season is now estimated to be $130k less than planned since it was purchased in late December 2008 after oil prices dropped. The cost of fuel in FY2011, the final installation season, will be purchased in one year. As a result of the lower than planned FY2010 fuel cost and fuel consumption in FY2009 that is less than baseline plans, the potential fuel cost exposure of $600k was removed from the contingency table below. This will be reassessed during the coming year. Item Risk Assessment & Potential Contingency Adjustments 1. Contingency likely to be required to mitigate technical, cost and schedule risks associated with the approved scope of work. Risk assessments are made at WBS-Level 4 to determine the value of the risk exposure as a percent of the cost of work remaining. 2. Pre-Operations for additional enhancements to the data storage and network systems both at the South Pole and in the Northern Hemisphere. 3. RPSC FY08 Closeout. The closeout is expected to return of RPSC authorized budget to IceCube contingency resulting from lower labor use and reduced support charges. 4. Annual estimate-to-complete for the remainder of the project for all elements of the work scope except RPSC support. This planning process is nearly complete. 5. Estimate cost of the limited amount of remaining instrumentation for an 86 string detector, 80 strings from the original proposal plus the six deep core strings financed by European partners. The additional instrumentation is five surface-to-dom cables and ~100 DOMs. 6. Current RPSC estimate of base cost to support the installation of six strings during the 2010/11 drilling season. The current RPSC baseline includes support for 80 strings. 7. The estimated cost to retro IceCube equipment/materials from the South Pole at the end of the project in FY2012. 8. Potential cost exposure should a full FY2011 drilling season be required resulting in the need to send UW personnel to the South Pole in FY2012 to dismantle the drill equipment/materials and prepare the equipment for retro out of South Pole. Total Available Contingency as of Nov 30, 2008 Estimate ($K) $2,405K $500K -$400K $1,000K $1,350K $1,340K $825K $450K $7,470K $8,649K Drill Operation and Installation Drilling at the South Pole continues ahead of schedule. Very minor problems were encountered, but all have been resolved successfully. Firn drilling for the last hole of the season was completed the week of January 12, 2009. By the end of December 2008, seven days ahead of schedule, ten new strings were deployed. As of January 15th, a total of 16 strings were successfully deployed including the deep core prototype string. Efficiencies in drilling resulted in less fuel use than originally planned. NSF approved continuing drilling and installation of up to 19 strings. It is now anticipated that the 19 th string will be installed by January 22 nd. The drilling crews will then proceed with camp winterization and depart South Pole about one week ahead of schedule. Digital Optical Module and Cable Production Status and Plans DOM Integration ceased as planned at all production sites pending the delivery of materials in November and December. It will resume in January 2009 at PSL and in Sweden. DESY in Germany successfully completed Page 4
all of their DOM Integration and Test activities on schedule. DOM production is not a critical path activity. Detector Commissioning and Verification Anna Franckowiak (Humboldt/Berlin) and Stephanie Hickford (Canterbury/Christchurch) were trained during this month and previous months for their upcoming trip to the Pole. They will be responsible for high-level commissioning of the new strings being deployed, and for taking the data so that we can verify the timing and calibrate the geometry of those strings. AMANDA will conclude operations at the end of the IC-40 run in March 2009. Data Acquisition Hardware and Software The IceCube clock system had a firmware bug because there was a leap second in a leap year which caused the clock to register two instead of one leap second and has caused an offset of 1 second relative to UT. The problem has been fixed but we are shopping around for another unit since this is the second time we have encountered a bug like this in the past 4 years. IceCube Maintenance and Operations Detector Up Time 97.3% IceCube Clean Run Up-Time not including AMANDA Array 94.5% Unscheduled Downtime 1.49% Events from DAQ 3.76 billion Data taking during Dec 2008 achieved an overall 97.3% uptime. The in-ice clean run uptime was 94.5%, lower than typical but high considering the large amount of activities and special runs/configurations as the pole season unfolds. SPS is being maintained and upgraded and caused some loss of uptime. On Dec 4th special runs were taken to develop and successfully test the Soft Local Coincidence pdaq feature, planned for the next running cycle to begin in April. On Dec 9th and 12th data taking was interrupted for about five and three hours respectively by failure of multiple drives on a new server for the secondary event builder. In the midnight between Dec 30 and 31st the IceCube GPS produced a wrong leap second. It was caused by the combination of the leap year (366 days in 2008) with the leap second that was accrued in the transition to the new year (which was indeed accrued in addition to the wrong additional one). As a result, on Dec 31st the AMANDA and IceCube clocks went out of phase and the overall detector rates increased. This issue is now resolved. Data Handling South Pole systems in the ICL began IC-40 physics operation in April 2008. Operations continued with systems operating relatively smoothly. The figure below shows the daily satellite data transfer rates. Page 5
The Data Handling work in December 2008 consisted of: 1) new hardware installed at South Pole for DAQ, processing and filtering, calibration and verification, and data handling and archiving; 2) additional DOM Hubs installed to support strings and IceTop tanks installed in the 2008/2009 season; 3) uninterruptible power supplies added and configured at the Pole to support additional computing; 4) new HSM tape based storage system installed at IceCube Data Center. Performance tuning is ongoing; and, 5) Operating Systems upgraded on the SPS system. Filtering, Software & Database Filtering of IC40 data began on April 14, 2008 representing the full transition from IC-22 to IC-40 running. The Joint-Event Builder and Processing and Filter server (JEB/PnF) merging of IceCube-40 and Amanda TWR data streams continues at South Pole sending filtered data sets of about 35 GB/day or more over the satellite. The major work in online filtering and software systems included: 1) new JEB release supporting: TWR+I3 flashing, new IceTop MB readouts and a filtering for them; 2) new Database tools that simplify the loading of information at pole, and mirroring of the DB in the North. These replace the existing set and have more uniform interfaces across all scripts; and, 3) Icetray V3 development work moving forward nicely. Anticipated release of this final version for end of project year is on schedule. Offline filtering of the IC40 data made significant progress last month. Accomplishments include: Finished standard processing script or for IC40 processing including all input from the physics working groups. Release of the standard processing meta-project (Version 3.0.0) for IC40 processing. Started mass production of high level reconstructions at the IceCube datacenter in UW. IC40 data tapes arrived from South Pole. Preparation and tests for re-filtering the early month of the IC40 run to take advantage of improved calibrations. Page 6
Simulation IceSim MonteCarlo event simulation program is fairly mature now, and further development is somewhat limited to improved performance and fine tuning. Some details include: The TWR simulation has been adjusted to the new DMADD thresholds measurements from the pole. Significant work on cleaning up the PhotonicsService project, and added a more efficient calling sequence for use with photorec, following the findings and recommendation of the code review. Planning to improve memory economy by merging the detector simulation modules. This will be a major change. Initial implementing of code has begun, with delivery for first testing in about a month. There is also a plan to use the same threshold treatment for IceTop and InIce. In addition to work on the simulation itself, there was significant progress on simulation production and filtering of simulation data. The main disk in the data warehouse /data/sim is read-only because it is full, and we cannot currently write data on it. This is due to the transition work in the date warehouse upgrade that is ongoing. We have temporarily used another available disk. We had planned to proceed with simulation data filtering rather than increase the volume of trigger level data files on disk for the moment. However, the work in December was severely hampered because of limited data availability during the date warehouse enhancements. Education and Outreach IceCube participated in the IPY's "Above the Poles" with a live event on December 4, 2008 (http://www.polartrec.com/live-from-ipy/archive). We also updated our IPY website for this date. Scientist, Mark Krasberg continued his Ice Stories Dispatches for the Exploratorium in December including a webcast from the South Pole on December 18, 2008 (http://icestories.exploratorium.edu/dispatches/index.php). Tom Gaisser, IceCube spokesperson, was interviewed on Discovery Channel Canada (Daily Planet) on December 22, 2008. When broadcast, the program will feature video of IceCube activities at the South Pole and an animation of the hot water drilling. Quality Assurance and Safety No significant issues to report. There are no reportable injuries so far this season. There are a few incident reports. The monthly reports are posted at IceCube Monthly Reports. The On-Ice Weekly Construction reports are posted at On-Ice Weekly Construction Report. Page 7
Prince Albert of Monaco is signing a DOM on the 16th of the season, the 56th overall. This string is the 1st of the six Deep Core strings and was deployed on Wednesday, January 14th. Photo by: Sven Lidstrom. Meetings and Events IceCube Software & Computing Review at UW-Madison Season Review Meeting at UW-Madison IceCube Maintenance & Operations Proposal Due Spring Collaboration Meeting at UW-Madison NSF M&O Review at UW-Madison Science Advisory Committee Meeting at UW-Madison NSF Annual Review Proposed Date Page 8 March 24 25, 2009 March 25, 2009 April 1, 2009 April 28 May 2, 2009 TBD May 20 May 21, 2009 June 3 4, 2009