OpenSFS News

Lustre Transparency and Momentum for 2014

By jfranklin

2014-Wallpaper

OpenSFS prides itself on being deeply committed to the philosophy of open source development. We bring together a wide range of organizations and individuals to solve problems that move open source file system development forward. Our “hands-on” working group structure means that OpenSFS Participants get deeply involved and can make a real difference.

In the spirit of open source, we want to provide greater visibility into OpenSFS inner workings as well. Please look for us to provide updates on OpenSFS discussions and progress as we move forward in 2014. As always, comments and questions are welcome.

For our first 2014 post, we wanted to take a quick look back on the accomplishments of last year. It was a big year for us! Lustre made some astounding strides forward.

In 2013, OpenSFS added two new Participants (Fujitsu and NCSA) while Intel moved up to Promoter level and joined the OpenSFS board. OpenSFS also pushed enhancements to our communications and marketing efforts, added more mature management infrastructure, continued funding the Lustre tree contract and added more Lustre development funding, supported enhanced feature availability, and had a significant hand in building a stronger foundation that the Lustre vendor ecosystem relies on for their product lines.

In particular, we continued to contribute financially to Lustre. We spent over $3 million on Lustre technical development and maintenance in 2013, our largest annual commitment to date. The result is that Lustre is healthier than ever, adding features that will directly benefit HPC, Big Data, and Enterprise users.

Highlights from 2013

·       Major releases of Lustre 2.4 and 2.5: Lustre 2.4 brought a new set of features that extend the capabilities of Lustre significantly, including Distributed Namespace (DNE), also known as “Clustered Metadata Servers,” providing horizontal scalability of metadata performance. Lustre 2.5 featured the highly anticipated Hierarchal Storage Management (HSM) functionality, often a core requirement in Enterprise and Big Data environments.

·      Opened the Community Lustre Portal at lustre.opensfs.org: Resources for Lustre developers now available in one convenient spot.

·      First Annual APAC LUG13 (Asia-Pacific Lustre User Group 2013): OpenSFS hosted the inaugural APAC LUG events held first in Beijing, China, and then in Tokyo, Japan. Attendance was exceptional and marked a big accomplishment for OpenSFS and the Lustre community, building better connections out to the Lustre community worldwide.

·      Largest LUG13 (Lustre User Group 2013): The largest attended LUG so far, Lustre users and developers had many reasons to celebrate with the announcement of multiple Lustre installations now breaking the 1TB/sec data access speed barrier (Fujitsu’s K computer and NCSA’s Bluewaters) and on-going announcements of multiple new Lustre installations worldwide. The next LUG 2014 will be held in Miami, Florida, from April 8-10. For more information please see: www.opensfs.org/lug14/

·      Welcomed our new Community Representative Director, Tommy Minyard, director of Advanced Computing Systems at the Texas Advanced Computing Center (TACC). The position is re-elected annually in April, if you are an Adopter or Supporter level OpenSFS Participant and want to nominate the next individual, please let us know.

·      Big presence at SC13 (Supercomputing Conference 2013): Held in the beautiful Denver, Colorado, OpenSFS sponsored the popular Lustre Birds of a Feather (BoF) session, displayed new Lustre Testimonials in our booth, and made face-to-face connections with a wide range of Lustre developers and users.

·      Continued Partnership with EOFS at the ISC13 (International Supercomputing 2013): OpenSFS organized a popular Lustre BoF focused on European issues, sponsored a breakfast panel with multiple Lustre experts explaining the expansion of the Lustre file system and its growth past traditional HPC into the broader Big Data space, and, of course, spent much time on the show floor talking Lustre with developers and users.

·      LAD13 (Lustre Administrators and Developers 2013) workshop: Sometimes known as the “Europe LUG,” our partnership with EOFS continues to be a key activity, connecting European users and vendors with their North American counterparts.

·      On-going Lustre development: OpenSFS continues to work on managing community testing under a contract with Indiana University, as well as hosting an open system for Lustre development and testing.

Expectations for 2014

2013 was a big year for Lustre, but we are not slowing down. We are starting off 2014 with great momentum in membership and funding. The Lustre User Group (LUG) 2014 will be our biggest ever. And new Lustre releases will push Lustre even more into the limelight.

Our priorities this year focus on our commitment to the Lustre community. We will continue to promote, protect, and advance Lustre through direct funding of Lustre development, sponsorship and management of community events, and Lustre community coordination. Fostering broader community development and community stewardship of Lustre releases is key.

Lustre could make a major difference in your industry this year. Join us!

We wish you the best for 2014 and look forward to your continued support and involvement in the Lustre community!

Video: Moving Lustre Forward

By jfranklin

In this video from the DDN User Meeting at SC13, Brent Gorda from the Intel High Performance Data Division presents: “Moving Lustre Forward: Status & Roadmap.”

OpenSFS at SC13

By jfranklin

SC13 - DenverOpenSFS was at Supercomputing 2013 November 17-20 in Denver Colorado. This was the 25th annual Supercomputing conference and exhibition, and the OpenSFS organization was  on the exhibition floor in booth 4125.

Among the activities at the booth were a drawing for Kindle Fire, which drew over 200 entries, and the OpenSFS Passport, a treasure hunt that rewarded visits to all the  OpenSFS partners in return for possibly winning an iPad mini. The Passport program had 37 participants.

But the high point was definitely the booth party, which gave a good number of folks the chance to meet and talk to some the Lustre developers and OpenSFS board members.
Details are at the SC13 events page.
SC13 -OpenSFS Booth

 

Photos by Jen Franklin

 

Lustre User Group 2014 Call for Presentations

By jfranklin

The 12th annual Lustre® User Group (LUG) conference, the primary event for technical presentations and discussions on the Lustre parallel file system and other open source file system technologies, is being held in Miami, Florida, from April 8-10, 2014. Doing something interesting with Lustre? Pushing the boundaries? We’d love to have you involved!

See our LUG 2014 Presentations page for details.

Lustre 2.5.0 Released

By jfranklin

The Lustre® Community Development Working Group (CDWG) is pleased to announce that the Lustre 2.5.0 Release is now available for download. The source code is also available from git.
This major release includes a number of new features:

  • Hierarchical Storage Management (HSM) (LU-3608) – allows data to be transferred between different storage types
  • Extended Attribute Cache (Xattr Cache) (LU-2869) – improved performance when reading extended attributes

Fuller details can be found in the change log, the scope statement and the test matrix. The following are known issues in the Lustre 2.5.0 Release:

  • LU-3833 : HSM does not check whether a file is sparse before archiving which could lead to unexpected results
  • LU-4152 : Under rare access patterns while re-exporting NFS it is possible that MDT threads might block until external action is performed on a file
  • LU-4159 : Lustre 2.4.x clients are unable to access a released file

Work is in progress for these issues and it is likely that these will be addressed in the 2.5.1 release scheduled for Q1 2014.

Please log any issues found in this release at the issue tracking website

We would like to thank OpenSFS for their contributions towards the cost of this release, and also to all Lustre community members who have contributed to the release with code and/or testing.

— October 30, 2013

Japan LUG 2013 Papers Posted

By jfranklin

Over the past few years, the annual Lustre® User Group (LUG) has been the main event worldwide concerning Lustre, bringing together Lustre developers, solution vendors, and Lustre users. OpenSFS has been organizing and supporting Lustre User Group meetings in the US for the past few years. With events in Japan and China, OpenSFS aims at better connecting Lustre users and developers in APAC to the worldwide open source Lustre community.

The Japan workshop provided existing and future Lustre users with a broad overview of the Lustre file system and its application in high-performance computing, analytics, and cloud environments. The event was free-of-charge.

For more details, the event report, and a full agenda and presentations, please see the Japan LUG 2013 event page.

China LUG 2013 Papers Posted

By jfranklin

Over the past few years, the annual Lustre® User Group (LUG) has been the main event worldwide concerning Lustre, bringing together Lustre developers, solution vendors, and Lustre users. OpenSFS has been organizing and supporting Lustre User Group meetings in the US for the past few years. With events in Japan and China, OpenSFS aims at better connecting Lustre users and developers in APAC to the worldwide open source Lustre community.

More than 80 attendees participated in the China LUG event, which provided existing and future Lustre users with a broad overview of the Lustre file system and its application in high-performance computing, analytics, and cloud environments. The event was free to attend.

For more details, the event report, and a full agenda and presentations, please see the China LUG 2013 event page.

RFP – W4570

By jfranklin

OpenSFS is pleased to announce a request for proposals for Lustre feature development, parallel file system tools, addressing Lustre technical debt, and parallel file system incubators (Solicitation Number W4570).

Our goals for this RFP are to:
1) Further the Lustre roadmap to meet the highest priority requirements defined by the community
2) Develop production quality tools to ease administration and use of open source scalable file systems
3) Address Lustre technical debt to improve the code base and documentation thereof
4) Encourage new efforts in open source scalable file systems for high performance and data intensive computing to broaden the set of solutions available to the community

To meet these goals we encourage innovative ideas and strong proposals that address one or more areas of interest detailed in the RFP. OpenSFS may fund multiple proposals as part of this RFP and we encourage broad participation from the entire open source scalable file system development communities.

Interested offerors are advised to review the RFP documentation below.

RFP Components:

– Proposal Preparation Instructions opensfs-proposal-preparation-instructions-final
– Lustre feature development Technical Specification technical-specification-lustre-features-final
– Parallel File System Tools, Addressing Lustre Technical Debt, and Parallel File System Incubators Technical Specification technical-specification-tools-final
– Example Statement of Work example-sow-final
– Sample Contract OpenSFS Contract-SAMPLE
– Evaluation Criteria opensfs-evaluation-criteria-final
– Terms and Conditions terms-and-conditions-final

Interested parties are encouraged to indicate their intent to submit a proposal no later than March 8th, 2013, via email to proposals@lists.opensfs.org. Please note that this is a controlled access email list and only members of the review committee will receive communications to this email address. Offerors should not attempt to subscribe to this email list. All postings to this email list are moderated.

Full proposals are due April 5th, 2013 at 11:59 PM EDT as detailed in the Proposal Preparation Instructions.

Interested parties must submit all communication (questions, comments, etc.) to proposals@lists.opensfs.org. OpenSFS will then post questions and our answers to the OpenSFS website.

While the RFP is open and under review, OpenSFS will not discuss the RFP except through the emails sent to proposals@lists.opensfs.org and responses on the OpenSFS website. Regular business activities that do not pertain to this RFP will be conducted in the normal manner.

All trademarks are the property of their respective owners.