Concurrent Threading Java Help

[oplin]

Anybody know how to execute a bunch of threads and then wait for them to finish or only execute 2 -8 threads while going through a while loop and then wait until the last item has finished running? There are roughly 40 iterations of the loop that take about 6minutes to process and cleanse 300megs of data and predicting a time based on megs won't work too well because this is run on different servers with different specs. Here is what i've got so far. That's pretty much the jist of it, i think i'm using the concurrent classes wrong and the executor stuff wrong. Any insite on this or a link that shows how to do this with an external file would be most helpful. If anyone wants all of the source code let me know but i can't give out any test files as the data is sensitive. These are 2 seperate classes also.

ExecutorService executor = Executors.newFixedThreadPool(3);
long waitTime = 360000; //6 minutes to wait for

while (iter.hasNext()) {
projectid = iter.next();
TextDataThread tdt = new TextDataThread(projectid, fileList, inputProjectRefs, fileout, ct);
executor.execute(tdt);
}
try {
Thread.sleep(waitTime);
executor.shutdown();
executor.awaitTermination
(waitTime, TimeUnit.MILLISECONDS);
} catch (InterruptedException ignored) {
}

//**** THe TextDataThread class parts
public TextDataThread(String projectidIn, FileList fileListIn, ConcurrentMap<String, ProjRefData> inputProjectRefsIn, FileOutputStream fileoutIn, CountDownLatch ctIn) {
//setters
}

public void run() {
ConcurrentMap<Double, Factmbr> memberList = new ConcurrentHashMap<Double, Factmbr>();
}

 

[Nemezer]

I don't think what you are asking for is any harder then the "java.lang.Object.wait()" and the "java.lang.Object.notify()".

Cheers!
Simon.

 

[kschneid]

What you've got looks basically correct, although I'd do away with the call to sleep. The call to awaitTermination will return a boolean so that you can determine whether or not the executor has been terminated within the provided timelimit (you can also call ExecutorService.isTerminated). If it hasn't, you could always decide to wait some more (or loop).

There are probably a few other ways to handle this, but take a look at the Javadoc for java.util.concurrent.CountDownLatch and see if that meets your needs. The second code example (Driver2) illustrates combining a latch with an executor.

 

[oplin]

Thanks for the tips, i tried the countdown thing that looks like a good way to do it but when i do countdown.await(); some reason it doesn't wait there it just goes through it, so i wrote this block of code. and it works just fine. When i write an Exectutor like Executor executor = Executors.newFixedThreadPool(4); when i loop through with executor.execute(blahblah); it will goto the bottom of the loop right away (but it does look like it queues up the threads so only 3 run at once,but i can't get it to wait for that either, It always waits for waitTime, unless i need to through this into a loop or something or a loop with sleep, if anyone comes up with a better way let me know otherwise i'm going to keep track of the threads and check to see when they are finished.
try {
//executor.shutdown();
executor.awaitTermination
(waitTime, TimeUnit.MILLISECONDS);
logger.warn("watingfor threads to stop executing");
} catch (InterruptedException ignored) { }

// ****** Below code works.
while(true) {
iter = threads.keySet().iterator();
while(iter.hasNext()) {
projectid = iter.next();
Thread temp = threads.get(projectid);
if (temp.isAlive()) {
logger.warn(temp.isAlive());
} else {
threads.remove(projectid);
}
}
Thread.sleep(10000);
if (threads.isEmpty()) {
break;
}
}

 

[oplin]

Got it working, i had a countdown somewhere in the code hidden away from playing around with this from yesterday. It's working with the fixed number of threads and the countdown thing. Thanks for the help and the link.

Now if anyone can let me know if this statement is the right way to write to a file when multiple threads are hitting it. Bassically it is a couple hundred throusands lines and i think this is the most efficient way to write to the fie, by sharing the fileoutputstream, Other than making buffout a static global variable.

private synchronized void writeFile(byte[] buffoutIn) {
byte buffout[] = new byte[2048];
buffout = buffoutIn;
try {
fileout.write(buffout);
} catch (FileNotFoundException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
}

 

[generelz]

Hi there.

byte buffout[] = new byte[2048];
buffout = buffoutIn;

The above two lines serve no purpose other than to slow your program down. You are allocating a new byte array which is referenced by the "buffout" variable and then re-assigning that reference right after it. You should just remove the two lines and use "buffoutln" instead. What exactly are you afraid of? Do you understand the implications of the "synchronized" keyword?

In general, if you want to see if a thread is "finished" and you have a reference to that thread, call the "join" method on a reference to that thread. The join method will return immediately if the thread has finished, or will block the calling thread until the thread finishes.

For example:

Thread myThread = new Thread(){
 public void run() { /*do some stuff*/ };
}
long startTime = System.currentTimeMillis();
myThread.start();
//wait for it to finish
myThread.join();
System.out.println("Thread finished in " + System.currentTimeMillis()-startTime + "ms.");

Was there a specific reason (other than not knowing any better) why you were using the more advanced concurrency libraries provided by Java? In this case I think a simple loop over the threads calling join on each would suffice.

 

[oplin]

Yeah pretty much a first go and didn't see much help on it.

Although executing all threads at once kills performance, so we like to setup config files so we can pass in the number of threads each server is capable of processing at once, or if we need to reduce the cpu of the program. I updated the program and it is running alot better now with error catching and it can tell if a thread gets stuck. Works very nicely now, reason for some of the code being messy is i was just given a 10000 line program written pretty much top down and was told to speed it it up.

If anyone has a good article on using StringBuffer instead of String let me know. Otherwise thanks for all of your help.

 

[kschneid]

If anyone has a good article on using StringBuffer instead of String let me know. Otherwise thanks for all of your help.

It should be pretty easy to find that sort of thing (like this), it's a popular subject. With 1.5, you can also use StringBuilder instead of StringBuffer (providing you don't need the thread-safety of StringBuffer).