--- All budget data must be uploaded into the Budget Input system by close of business Aug. 1 – no exceptions.  In addition, all Effort Requests (SWF) should be uploaded by managers by June 30.

Tactical plans should be completed by Aug. 15. (CCD’s deadline for the first draft is June 30. This will give us time to properly formulate our plans and provide input to the overall Computing budget.)

--- Submit your self-evaluations
Self-evaluations should be submitted to your manager for review no later than July 5. See a complete timetable for performance and salary review.

--- Off-site training regulations
Attending off-site training?  DOE conference regulations apply to all business-related travel, even when it is local or an entity other than the laboratory is paying for it. Individuals must obtain proper approval in advance. See your administrative support staff to prepare Travel Authorization forms. Failure to do so may result in expenses (including registration fees) not being reimbursed.

---The design manager from Potbelly’s Sandwich Shop contacted Fermilab regarding hanging some historic photos of Fermilab on the walls of the Warrenville shop. Fermilab Archivist Valerie Higgins worked with Adrienne Kolb, Fermilab Historian, and Kurt Riesselmann from the Office of Communication to find appropriate photos. She selected two of them.

--- A photo from the DOE archives of the Cockcroft–Walton generator was featured with a caption by Mark Kaletka in this Gizmodo UK article.

---Ruth Pordes gave a presentation about Fermilab scientific computing to about 100 members of the Sun City Huntley Computer Club on June 6. View the presentation. (Ruth’s talk begins at about 14 minutes.)

--- From Bakken’s weekly newsletter:

This piece on the computing infrastructure needed to handle the huge amounts of data generated by the  LHC appeared in several computing magazines. 
It mentions Fermilab as a Tier-1 computing center for the CMS  experiment. With Fermilab leading the way, the U.S. handles 35 percent of the worldwide computing power for CMS.

CMS at Fermilab hit a new record for end-to-end traffic the week of June 8, reaching 75 Gbits/sec over our 100 Gbit ESnet link. The previous record was ~40 Gbit/sec. These kinds of transatlantic rates don’t just happen – but are the result of sophisticated planning, deployment and execution. Congratulations to the networking crew for providing the underlying framework that allows the CMS Tier-1 Center to remain ‘the place’ in the US to work on LHC physics!

The WireCAP research project, led by Wenji Wu and Phil Demar, is listed in the 2015 National Innovation Summit showcase.

--- MU2E is running a big Monte Carlo campaign on the Open Science Grid. The plot below shows job efficiency on various OSG sites. The size of the area represents the amount of wallclock-hour jobs spent on a particular site. Some of the inefficiency is due to sites pre-empting the jobs. Click on image to view a larger version. Image courtesy of Tanya Levshina. 


New employees

Mohammed Yahya (CCD/Information Resources/Content Management)
Paul Tader (SCD/Scientific Facilities/Distributed Computing Services Operations)
Laura Fields (SCD/Scientific Programs)
Hannah Ward (OCIO/Service Management/Communications)
Lindsey Gray (SCD/Scientific Programs)

May Anniversaries
(5, 10, 15 & 20+ years)

Adam Walters - 35 years
Jean Reising - 31 years
Marc Mengel - 24 years
Eric Neilsen Jr. - 24 years
Roger Slisz Jr. -24 years
Matt Crawford - 23 years
Burt Holzman - 10 years
Andrew Norman - 5 years
Albert Rossi- 5 years

June Anniversaries
(5, 10, 15 & 20+ years)

Keith Coiley - 46 years
Etta Burns - 33 years
Nanette Larson - 33 years
Andrew Romero - 28 years
Bob Andree - 27 years
Jo Ann Larson - 26 years
Lisa Giacchetti - 25 years
Dmitry Litvintsev - 15 years
Oliver Gutsche - 10 years
Wenji Wu - 10 years

Spot Award Recipients

The Spot Award program’s purpose is to encourage and recognize exceptional contributions by individuals and teams with “on-the-spot” rewards at the time of achievement.

Congratulations to the following Spot award recipients since the inception of the award program:


John Galvan
Rachel Hurd
Carla Lloyd
Jemize Ortiz
Luann O'Boyle
Bernadette Tabor
Marcia Teckenbrock
Kevin True


Patricia Cameron
Geoffrey Cluts
Keith Coiley
Penelope Constanta
Pawel Grawender
Maung Han
Valerie Higgins
Rick Hill
Norman Ho
Adrienne Kolb
Paul Lauss
Briant Lawson
Lauie Loebel-Carpenter
Laura Mengel
Kimberly Myles
Keenan Newton
Sriram Sankaranayanan
Joseph Syu


Dennis Box
Michael Diesburg
Stu Fuess
Patrick Gartung
Bonnie  King
Marc Mengel
Gene Oleynik

Marc Paterno
Pat Riehecky
Mark Schmitz
Neha Sharma
Anthony Tiradani

Two SCD physicists, 48 hours, more than 2000 miles and one electric car
Bo Jayatilaka driving a Tesla
Image courtesy of Sam Paakkonen

Last month, Bo Jayatilaka, an SCD physicist, chose an unprecedented way of traveling from his home in Boston to the FabrIc for Frontier Experiments (FIFE) workshop at Fermilab.  Jayatilaka traveled with fellow SCD physicist, Michael Kirby, their friend, Tom Rammer, and two others on a trip to Fermilab that lasted 48 hours, required a route through 12 states and part of Canada and took place in one brand-new Tesla Model S, an electric car. After winning a 48-hour Tesla loan at an auction, Rammer asked Kirby to help him come up with the best way to take advantage of the opportunity. The pair decided to drive the Tesla from Chicago to Boston and then back to Chicago to take Jayatilaka to the FIFE workshop. They recruited a film crew to follow and record their adventures between charging stations to and from the East coast. Over the course of their road trip, they discovered both the impressive power of the Tesla and the unfortunately limited infrastructure that keeps electric cars from becoming more popular. Read more here.

Tips of the month

Service Desk: After-hours outage notifications

This template (stored in FermiPoint) should be used in the event of a service outage that requires notification and occurs during non-business hours. Business hours are Monday through Friday, 8 a.m. to 5 p.m. During business hours, the regular Service Desk communication process should be used.

FermiPoint: 10 minute write lock on files

When a contributor edits a document checked out to them in the client (Word, Excel etc.), a “File is locked for editing by another user” message may appear preventing them from making edits. This occurs due to a “write lock” placed on the document by the system so that no other user can edit the file at that time. This write lock lasts 10 minutes and can also occur when a file closes unexpectedly and the contributor attempts to re-open and edit it, a check-in was unsuccessful, or there was a lost network connection. It is recommended to wait 10 minutes, and then attempt to edit the document again- it should be editable.   Also, after 10 minutes, the contributor can check in the document to let other contributors check it out and edit it.

For more information on write locks in SharePoint, please see this Microsoft article.

From the CIO: Benching Ourselves in the Pacific Northwest
Rob Roser
Chief Information Officer Rob Roser

Every once in a while, it’s good to get a benchmark and see how well we are doing.  That opportunity happened a few weeks ago when about a dozen of us traveled to Seattle for the annual National Laboratories Information Technology Summit (NLIT).   This meeting brought together IT folks from all of the national labs to share ideas, best practices, and learn from each other.  A number of vendors, taking advantage of the co-location of all these experts, came as well to showcase their wares.

The level of talks at NLIT was really quite good.  Talks covered the very broad spectrum of IT experiences at the various labs including service management ideas (as well as the tool, Service Now), IT strategy, moving more into the cloud, Voice over IP (VOIP), Workday, and single-sign-on authentication, just to name a few of the topics. 

While I went to some of the Fermilab talks, I tended to focus on other speakers to see what other labs were doing and consider whether there are things we should be doing differently.   There are a few takeaway themes that I would like to share here.

First, we are doing very well as an organization, and we stand up well to the best IT shops out there.  In many areas, we are out in front—pushing the technology where others are still trying to find a clear path.  A few examples include FermiWorks, the services approach to IT and cybersecurity, though there are many other examples as well.  There was always a crowd around our speakers of people seeking to learn more.

Second, we are working on the right things.   I feel that we are on the “leading edge” of innovation, but not the “bleeding edge.”  Namely, we are adopting new products once they are sufficiently mature that it makes sense to do this.

Finally, we have an excellent team here at the lab.  I spoke to a lot of people at the conference, and I realized how lucky I am to have this team.

The other benefit of NLIT for me is that I first got to meet DOE’s new CIO, Michael Johnson.   DOE is on its third CIO since I took the reins.  Yikes.   Michael gave the closing talk at the conference and then spent a day in a meeting with all of the lab CIO’s behind closed doors.  I walked away from our meetings very impressed.  He is very smart and has quickly picked up what the main issues are.   Given he is a political appointee, he will only be in this role a short while, and thus has to pick carefully how he wants to make an impact.  His goal is to improve how the labs and DOE work together in tackling IT and cybersecurity problems.  He is trying to streamline his organization and add a level of transparency and trust that goes both ways.   If he is successful, both the labs and DOE will benefit.  I am both hopeful and optimistic that he will be able to “steer the ship” in a positive direction and that the course will last well past his tenure.


Communities collaborate to take on next-generation technology
A Synergia particle-in-cell beam dynamics simulation.

Scientific Discovery through Advanced Computing (SciDAC) is a Department of Energy Office of Science program that funds projects with both advanced computing research and scientific elements. This encourages strong and effective partnerships between the Office of Advanced Scientific Computing Research and other research offices. Within each project, groups of different institutions then work together towards common goals.

The two main projects based at Fermilab are ComPASS, the Community Project for Accelerator Science and Simulation, led by Panagiotis Spentzouris, and “Searching for Physics beyond the Standard Model: Strongly Coupled Field Theories at the Intensity and Energy Frontiers,” which works on community software for Lattice Quantum Chromodynamics, led by Paul Mackenzie. SciDAC recently awarded both projects with funding for the next year.

The benefits of SciDAC funding are manifold, but two in particular stand out. “SciDAC establishes communities,” said Spentzouris, head of the Scientific Computing Division. “The intellectual environment this fosters and effort it supports allow ComPASS to develop powerful tools that are extremely useful in all aspects of accelerator design and operations.” Jim Amundson, head of the Scientific Software Infrastructure department added that this community provides Fermilab, a single-purpose laboratory, with invaluable connections to computer scientists and applied mathematicians.

Secondly, SciDAC awards help combat the perennial problem with technology: being up-to-date means you are already behind. SciDAC funding “allows software developers to meet the challenges of being able to run efficiently on next-generation machines,” said Jim Simone, head of the High Performance Parallel Computing Facilities department. Beyond this, they allow collaborations to “push the cutting edge of computing technology to serve the needs of science,” said Amundson.

We see SciDAC’s success in both Fermilab-led projects. ComPASS has developed a toolkit to model and understand particle accelerators as well as advanced accelerator concepts such as plasma wakefield acceleration. Within this, the collaboration has created Synergia, a beam dynamics framework which has achieved “what we think is the largest single beam dynamics simulation ever, 100,000 turns composed of over 7 million simulation steps,” said Amundson.

LQCD comparison graph
An LQCD comparison of the Conjugate Gradient matrix solver (red curve) and an eigenmode "deflation" method (green). Fewer iterations are needed with deflation. The sparse Dirac matrix in this example has 127,401,984 rows and an equal number of columns. Image courtesy of Jim Simone.

For Lattice QCD, Simone highlights that SciDAC has been “critical to getting the science done.” SciDAC code routinely achieves excellent performance on hundreds of GPUs and many-thousand core jobs on Fermilab clusters, and tens of thousands of cores on the current DOE leadership machines at Argonne, Oak Ridge and NERSC. The project will ready code for the next generation of leadership supercomputers that will exploit GPUs and the Intel MIC architecture to achieve even better performance.

By encouraging partnerships and enabling directed effort, SciDAC allows projects to stay abreast of technological advances and to drive innovative research.

~Clementine Jones

CCD Spotlight

Wei Gao
Information Systems, Enterprise Applications
Wei Gao

I joined Fermilab in 1997 with the title of computer professional. I am now an application developer & system analyst in the CCD Enterprise Applications group. I support the lab’s financial system, the Kronos Fermilab Time and Labor (FTL) system and FermiWorks. My work helps provide the capabilities for these systems to enhance business functionalities and services, and it helps management better understand laboratory operations and expenses.

I participated in several major implementations over the past 18 years. One of them was the project costing implementation, which made approvals and reporting more effective and efficient. Another significant project I worked on was the implementation of electronic timecards, which streamlined efforts from timecards to the lab’s financial system.

In addition to supporting system operations, such as bug fixes, enhancements and upgrades, I am currently working on several projects dedicated to enhancing the lab’s financial system. One of these projects is standard labor rate costing. Currently, labor costs use employee salary rates. With standard labor rate, it will be easier to predict actual costs for projects, it will help with budgeting and planning for operations and it will simplify cost estimates. Another project I am working on is the implementation of postdoc burden at different rates, a requirement for the DOE. I am also involved with implementing a new lab-wide budget planning system (BPS). The BPS will make budget and resource planning, as well as cost tracking and control, more efficient. Ultimately, it will enhance reporting capabilities.

With the rapid advancement of computing technology, my job often requires developing new skill sets. This has motivated me to learn new things and keep up to date with new technology. I love being challenged and working on new projects. The IT job today also requires a lot of teamwork. We have a great team, and we always work together to overcome obstacles in order to ensure the success of our projects. The work that we accomplish is very rewarding, and I look forward to continuing to be a contributor to this team.

SCD spotlight

Chris Jones
Systems for Scientific Applications/Scientific Software Infrastructure/Framework and Software Technology

Chris Jones

I have been working on the software used by the CMS experiment since 2005. I started by collaborating with the software development team at Fermilab to develop the CMS 'offline framework'. An 'offline framework' is the experiment's core software which is used to control all the non-data acquisition work needed by the experiment. This includes simulating the results of beams colliding in the detector, determining which particles traversed through the detector after a collision and supporting physicists' analysis of the data.

In 2009 I left Cornell University to join the Fermilab CMS software team. The following year, I became the CMS Level 2 manager for Core software which is the position I still hold today. In addition to being a developer of the 'offline framework', I am also the manager responsible for its development and maintenance. Recently, I was promoted to lead the new Framework and Software Technology group within the SSI department. In total, I have been designing, developing and maintaining 'offline frameworks' for nearly two decades.

For the past few years, I have been working on a major upgrade to the CMS framework. Not long ago, new computers were more powerful because their CPU ran at a faster rate. However that trend is no longer true and instead, new computers are becoming more powerful by using more CPUs. The CMS framework was originally designed to only use one CPU. My recent work has been to design and lead the implementation of the CMS framework to use multiple CPUs simultaneously through the use of operating system threads. We were able to complete this upgrade in time for it to be used for the just started LHC Run 2.  CMS is the first LHC experiment to utilize multiple operating system threads with ATLAS and LHCb still in the early phases of development.

-- Faculty from Chicago State University (CSU) and members of the National Society of Black Physicists visited Fermilab on Tuesday, May 12 as part of a larger collaboration meeting with some other institutions.  A discussion of “Computing and computer security” attended by Irwin Gaines, Joe Klemencic, Art Lee , Mark Kaletka and Ruth Pordes focused on cybersecurity and ways in which Fermilab could aid in CSU's course offerings leading to advanced certifications.  Following these talks, they were taken on a tour of the Grid Computing Center by Stu Fuess, David Mason, Ruth Pordes and Adam Walters

-- Members of the Enterprise Engineering Applications group visited ATI Forged Products in Milwaukee to give a presentation about how Fermilab has used AutoCAD integration with Team Center.  ATI had previously shared information about their experience with Team Center to better inform Fermilab’s Team Center implementation, and so this is a great example of the mutual benefits of information sharing.

-- SCD is exploring avenues for new partnerships. Adam Lyon, associate head of Systems for Scientific Applications, visited the Illinois Institute of Technology (IIT) Applied Math department in March.  Fred Hickernell, head of Applied Math, along with his student Lluís Antoni (Tony) Jiménez Rugama followed up with a visit to Fermilab.  Tony is interested in modern quasi-Monte Carlo methods, i.e., using quasi-random sequences or sub-random sequences for numerical integration, which may improve physics simulations. This summer, Tony will work with SCD’s Steve Mrenna exploring these techniques.

Another possible partnership is with the HDF Group, a non-profit organization that develops and maintains the HDF5 (Hierarchical Data Format) technology.  HDF5, which can handle extensive parallel i/o in multi-processor environments, is popular in high performance scientific computing and finance, but is relatively unknown in HEP.  HDF CEO Mike Folk and Senior Developer John Readey visited Fermilab with Glen Sato, a consultant.  Following up on discussions with SCD’s Scientific Software Infrastructure department, there is potential for a collaboration to try HDF5 with Muon g-2 data and explore the possibility of integration with ROOT.

-- On May 27, Leo Michelotti, from SCD’s Accelerator Simulation group, answered questions from a group of 31 high school students from Black River Public School in Holland, MI.

-- Now playing in the FCC lobby:

"Resources Analysis," Stu Fuess, Scientific Computing Portfolio Management Team (SC-PMT) Review, Fermilab, Illinois, March 5.

"Scientific Workflows Using Science Gateway Technology," Saba Sehrish, XSEDE Gateways and Workflows Symposium Series, March 13.

"Cybersecurity Continuous Monitoring at Fermilab," Irwin Gaines, NLIT, Seattle, WA, May 5.

"Federation at Fermilab," Al Lilianstrom, NLIT, Seattle, WA, May 5.