High Performance Computing and Big Data
Be it personalized medicine or a fully interconnected laboratory, we will face huge a amount of data to process and evaluate. This raises many challanges from a software and hardware perspective equally. A short and incomplete overview can be found in the following list.
Some Challanges
- Data privacy
- Programming models
- Energy efficiency
- [[ https://en.wikipedia.org/wiki/Amdahl%27s_law | Ahmdahl's law ]]
- Network topologies
- Load balancing
- Maintenance, availability and reliability
Local Learning Cluster
We should build a local cluster to play with different cluster setups. A very simple approach is to connect multiple raspberry pi's together, as shown in the following link. https://blog.hypriot.com/post/how-to-setup-rpi-docker-swarm/ This give a simple environment to learn the foundations of HPC with a homogenous cluster.
Frameworks
- [[ http://hadoop.apache.org/ | Java High Performance Computing (Hadoop) ]]
- [[ http://project-thrill.org/ | C++ High Performance Computing (Thrill) ]]
References
[[ https://pdfs.semanticscholar.org/9b6d/deb90dac8a828225bd58c9cf2f8ddc232812.pdf | Ashfaq A. Khokhar, Viktor K. Prasanna, Muhammad E. Shaaban and Cho-Li Wang. Heterogeneous Computing: Challenges and Opportunities. 1993 ]]
[[ http://journals.sagepub.com/doi/abs/10.1177/1094342015597083 | Al Geist, Daniel A. Reed. A survey of high-performance computing scaling challenges. 2015 ]]
