Log in

As a charitable Service based non-profit organization (NPO) educating and connecting the High Performance Computing user community to state of art technology for the purpose of Optimizing business processes and workforce advancement. Our technology focus includes AI, Cloud Computing, Data Science, Deep learning, Machine learning and visualization utilized in applications ranging from Manufacturing and Engineering, Financial services, Life Sciences, Energy, Academia and Government.

Lunch and Learn - March 2016 - Download Presentation! - Working with Multi-Terabyte Data Sets

30 Mar 2016 12:12 PM | SHPCP (Administrator)

You Need to Do What Now? Working with Multi-Terabyte Data Sets

Speaker: Dan Bedard, Executive Director

As multi-terabyte seismic data sets have become commonplace, new data management challenges have emerged: Processing takes time, especially if you have to repeat it. How do you mitigate the risk of hardware and file system failure? How do you ensure you're making efficient use of your storage infrastructure? How do you share data between workgroups while enforcing compliance with government regulations, licensing terms, and company data security policies? And how do you do this on the shifting sands of changing technology? 

No really. How? The first part of this presentation is intended to elicit information from you, experienced HPC professionals. We want to find out what approaches you've adopted to address this generation of data management problems. Then we're going to explore an ongoing proof of concept (POC) in which we're implementing a tiered storage system based on perpetually free open source software. Not only will the reference POC migrate data to the appropriate storage location based on usage history, but it will also set the stage for advanced data load balancing, rich metadata search, network-efficient sharing, and guaranteed chain of custody. Let's dig in to the management challenges of multi-terabyte data sets. 

The iRODS Consortium leads development and support of the Integrated Rule-Oriented Data System (iRODS), free open source software for data discovery, workflow automation, secure collaboration, and data virtualization. As an alliance of vendors and end users from data-intensive domains, the iRODS Consortium provides a production-ready iRODS distribution and professional integration services, training, and support. Top researchers in geosciences, life sciences, and information management use iRODS to control their data. Learn more at


Presenter’s Bio:

Dan Bedard, Executive Director
Dan oversees the strategic direction of the iRODS Consortium and leads member recruitment. Prior to joining the iRODS team, Dan managed programs and business development for an RF amplifier company, developed electronics in RENCI’s Innovations Lab, and designed devices for special projects for the United States Government. Dan’s personal interests include manned space flight and physical training for endurance and strength. He holds a Master of Science in Electrical Engineering from North Carolina State University and a Bachelor of Science in Electrical Engineering from the Massachusetts Institute of Technology (MIT).









Powered by Wild Apricot. Try our all-in-one platform for easy membership management