For my senior project, me and my group have implemented a distributed system for a company for managing devices on a network. It's been alot of fun, and I've learned alot. I'm going off to work now, but not on anything like that. I'd like to continue to learn about and deisgn and code systems like this. I'm having some problems coming up with ideas on projects to do to teach me. Anybody have any suggestions? Currently, the system we're writing is in java, but I'm open to anything. Java does a good job providing technology to do distributed apps, but I'm sure alot of languages are just as capable. Also, if there are any books I should look into buying to gain knowledge on the subject?
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Navigation
Install the app
How to install the app on iOS
Follow along with the video below to see how to install our site as a web app on your home screen.
Note: This feature may not be available in some browsers.
More options
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
You are using an out of date browser. It may not display this or other websites correctly.
You should upgrade or use an alternative browser.
You should upgrade or use an alternative browser.
Distributed Applications
- Thread starter Wingy
- Start date
What was it that interested you about the project your old group was doing? Why was it distributed? For redundancy? Geographical reasons? Uptime? Computing power?
I'm interested distributed computing for problems that are larger than one computer. Before I left Microsoft, I was thrilled to work on a distributed, scale-out database system. I did a bunch of research into scaling-out another product I had worked on, too. In both cases, redundancy was a secondary feature to allowing more work to happen than any one (sensibly priced) computer could handle.
At home, I've been playing with some similar problems:
Distributed Algorithms by Lynch is a seminal reference in the field, but it is starting to show its age. (ISBN 1558603484). In Search of Clusters by Pfister is really great, and a must-read if you like this area (ISBN 0138997098).
There are lots of papers on the subject, though they're almost always focused to your specific area of research. The Google File System is an interesting read, for example.
I'm interested distributed computing for problems that are larger than one computer. Before I left Microsoft, I was thrilled to work on a distributed, scale-out database system. I did a bunch of research into scaling-out another product I had worked on, too. In both cases, redundancy was a secondary feature to allowing more work to happen than any one (sensibly priced) computer could handle.
At home, I've been playing with some similar problems:
Say you have 4 billion integers. How can you find the median of the set? Ten billion? A hundred billion?- I found a site to download the first 70 billion digits of pi and e, both. What's the longest common substring of digits in both numbers?
- I like to play with Wikipedia's corpus. It's large in a snapshot, and absolutely overwhelming if you consider the revision history, too. How can you handle, store, load, decompress, manipulate, search, or render such a data set?
- Web searching. How do you make multiple machines crawl? How do they report and collect their findings? How can it be cataloged and searched?
Distributed Algorithms by Lynch is a seminal reference in the field, but it is starting to show its age. (ISBN 1558603484). In Search of Clusters by Pfister is really great, and a must-read if you like this area (ISBN 0138997098).
There are lots of papers on the subject, though they're almost always focused to your specific area of research. The Google File System is an interesting read, for example.
hokatichenci
Gawd
- Joined
- Oct 28, 2004
- Messages
- 722
Distributed systems are so wildly generic that it's really hard to know what a person means when they say distributed without some specific context. Then theres many ways to organize a distributed system that you really need to know a lot about the specific application.
What is it exactly that you want to do? There are so many different things to choose from, one of the things I was working on was distributed load balanced authentication mechanisms involving a checks/balances system.
Some things that might interest you:
Operating Systems - OpenMosix, Windows Advanced Server Cluster?
Networking File Systems - NFS, iSCSI, Ceph, PVFS2, Google Filesystem (theres papers but nothing you can toy with)
L3 Load Balancers - Linux Virtual Server (LVS)
Things to load balance - MySQL, Apache/PHP (session coherency)
Shared Memory Libraries - APR (Apache Portable Runtime)
Application remote procedure calls - Unix RPC, XML-RPC, MPI
The book we used in my distributed systems class was pretty good, though not bleeding edge by any means, and focused on the core concepts more than anything else. "Distributed Systems Principles and Paradigms" by Andrew S. Tanenbaum/Maarten van Steen.
If you are interested in writing secure systems at all, I would highly also suggest another book called "Network Security Private Communication in a Public World" by Charlie Kaufman/Radia Perlman/Mike Speciner. Alice and Bob are perfect targets in a distributed system. Its focus is applied cryptography but there is a lot of good things in there that helped me tighten up security on the distributed framework I was working with.
What is it exactly that you want to do? There are so many different things to choose from, one of the things I was working on was distributed load balanced authentication mechanisms involving a checks/balances system.
Some things that might interest you:
Operating Systems - OpenMosix, Windows Advanced Server Cluster?
Networking File Systems - NFS, iSCSI, Ceph, PVFS2, Google Filesystem (theres papers but nothing you can toy with)
L3 Load Balancers - Linux Virtual Server (LVS)
Things to load balance - MySQL, Apache/PHP (session coherency)
Shared Memory Libraries - APR (Apache Portable Runtime)
Application remote procedure calls - Unix RPC, XML-RPC, MPI
The book we used in my distributed systems class was pretty good, though not bleeding edge by any means, and focused on the core concepts more than anything else. "Distributed Systems Principles and Paradigms" by Andrew S. Tanenbaum/Maarten van Steen.
If you are interested in writing secure systems at all, I would highly also suggest another book called "Network Security Private Communication in a Public World" by Charlie Kaufman/Radia Perlman/Mike Speciner. Alice and Bob are perfect targets in a distributed system. Its focus is applied cryptography but there is a lot of good things in there that helped me tighten up security on the distributed framework I was working with.
Thanks for your replies guys.
mike, when you say that that book is getting aged, are there ways in which I could know what parts are aged, and which ones aren't? Or possibly follow it up with the next one you named, and also something like hokaticheni named. As with anything programing related, I understand the necessity of learning the fundementals first and go from there.
I'm kind of excited to look into some basic distributed algorithms... Minus some of the basic sorting alg's we learned here. Also there was a basic routing algorithm, the name escapes me. It seems like distributed algorithms have an entirely diferent mindset to go along with them alot of the time.
Thanks for your ideas and insight guys. I like the pi, e problem, and the others.
mike, when you say that that book is getting aged, are there ways in which I could know what parts are aged, and which ones aren't? Or possibly follow it up with the next one you named, and also something like hokaticheni named. As with anything programing related, I understand the necessity of learning the fundementals first and go from there.
I'm kind of excited to look into some basic distributed algorithms... Minus some of the basic sorting alg's we learned here. Also there was a basic routing algorithm, the name escapes me. It seems like distributed algorithms have an entirely diferent mindset to go along with them alot of the time.
Thanks for your ideas and insight guys. I like the pi, e problem, and the others.
I dunno. IIRC, the book is pretty well-referenced, so if you see a chapter about a topic, check the references. If the papers were pretty old, that indicates the topic is getting kinda stale. OTOH, if the papers are near the publication time of the book, maybe they're fresher. You'll also pick up on it as you have more experience. Learning the old way to solve something doesn't prevent you from also learning the new way to solve something later.
I'd start with In Search Of, anyway. It's a great overview of the problem space.
I'm pretty excited about pi-e digits, too. I've got a dynamic algorithm for finding longest common, and it doesn't take much memory -- but that just means 140 gigs of memory instead of 4900 terabytes of memory! I'll still have to do 4900 trillion operations, and combined with all the I/O my estimate for runtime on a single box is getting pretty lareg.
I'd start with In Search Of, anyway. It's a great overview of the problem space.
I'm pretty excited about pi-e digits, too. I've got a dynamic algorithm for finding longest common, and it doesn't take much memory -- but that just means 140 gigs of memory instead of 4900 terabytes of memory! I'll still have to do 4900 trillion operations, and combined with all the I/O my estimate for runtime on a single box is getting pretty lareg.