Yeah. I've got over 40 in the upload queue. Hopefully it gets fixed soon.
![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
-
Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.
WCG?
- Thread starter Toconator
- Start date
danwat1234
n00b
- Joined
- Sep 15, 2010
- Messages
- 13
Natalia, a senior username I recognize from SiDock; "We are recovering both SiDock@home and RakeSearch after a failure of both SSD discs in a mirror. The team is updating the server (hardware), and also the data (including the forum). It may take few weeks more... We will announce as soon as the projects work again." "Update: RakeSearch project is working, but its website is still under maintenance. SiDock@home is yet to be recovered." https://boinc.berkeley.edu/forum_thread.php?id=15504
- April 29, 2026
- BOINC traffic resumed around 12:30 UTC on April 25th, 2026 and cluster is currently stable as of this writing (18:00 UTC on April 29th, 2026) - the issues at the data center was resolved, recovery was successful and the issues causing large numbers of 404s on download and 503s on upload unrelated to the issue at the data center were both resolved as well. We implemented backpressure for the validators, juggled some services like the backfill validations off the database cluster nodes to other nodes in the cluster, further tuned postgres for our workload, and modified some BOINC components to harden the cluster against future outages.
- BOINC stats export to https://download.worldcommunitygrid.org/boinc/stats resumed - server status page will follow once ARP1 and MAM1 are up and running again.
- IN_PROGRESS results do not display on the website - until the credit_flusher batch upserts those rows into the legacy MariaDB database after validation, these workunits are not visible to the website APIs. We are working to add to the Results API a fetch and cache from the new BOINC database postgres cluster so that these IN_PROGRESS results can be seen on the website and retrieved from the APIs. Likely, this fix will conincide with improvements to allow users with Result sets large enough to timeout the API to see or at least download their results, and fixes and tooltips for the new Summary feature on the Results page.
- Data Sharing radio button does not work - thank you for the report, working to fix this.
- 403 Forbidden - frustrating forum users - when we fixed the team challenge registration, the issue causing 403 Forbidden on that page was the updated mod-security rules for apache2 on the load balancer server. We will start there and look at the mod-security rules from load balancer through to the container that hosts the website behind HAProxy which also has it's own set of rules, and hopefully provide relief soon.
- When will ARP1 be released? - current blocker is the geographical split with overlapping edges between regions to match our partitioned backend, so that downloads and uploads will be routed to a mostly contiguous geographic region of the overall sub-Saharan region for which the project is predicting the weather. As this involves fetching ARP1 results across boundaries of those mostly contiguous regions so that "halo" domains can have their next generations of workunit inputs generated from the completed work of all their neighbours within, and the neighbours across the partition border, it requires more devleopment and testing. Now that we are seeing stability in the new architecture, this is a priority and we will update as we get a better sense for the exact timing. We may release workunits that are completely within a geographic partition to test the ARP1 BOINC components in general, before we can announce that the project is back up and running.
- When will MAM1 and the GPU build "MAMG" workunits be released? - while MAM1 and the corresponding beta30 project are up to date with the MCM1 pipeline and could have work issued at any time, we are exploring the options afforded to us by upgrading our BOINC build such as the BOINC Universal Docker App "BUDA" (https://github.com/BOINC/boinc/wiki/BUDA-overview), and also using Mojo with the existing LibTorch support within the application to run on newer AMD and NVIDIA GPUs https://docs.modular.com/max/develop/custom-kernels-pytorch). We expect to start sending out batches of MAM1 workunits for the mt CPU build through the beta30 app and possibly the MAM1_9999900+ testing range this week, barring some new blocker, and will update on GPU support as we release new builds through the beta30 application.
- Joined
- Dec 27, 2015
- Messages
- 3,975
I haven't run WCG since the 2025 BOINC pent race where WCG was chosen as a marathon project. Even then the project barely held up during the pent, lol.
Just got back to town. I had one project aborted @ 100% but 3 threads running and a whack of projects waiting to start. I'm running BOINC 8.2.4 . all fine here
WCG Status Update 2026-08-21
Updates on the status of the Africa Rainfall Project (ARP1), Mapping Arthritis Markers (MAM1, beta30), and Mapping Cancer Markers (MCM1) projects at WCG. Updates on web and API development that is currently in progress will be provided next week.
ARP1
- ARP1 workunits are being distributed again, consider the project intermittent pending a full restart. ARP1 workunits are orders of magnitude larger than MAM1 and MCM1 workunits, so the in-memory cache across a partitioned BOINC backend that allowed multiple backend servers to share the load, and work around the slower NFS performance and database IO at Nibi compared to the dedicated appliance we had at Graham, is simply less beneficial for ARP1. Further, it negatively impacts MCM1 and MAM1 for little benefit, as we simply cannot hold enough ARP1 workunits resident in memory to make a dent in throughput anyways. We now have a working ARP1 lifecycle based on our NFS and block storage as we await access to object storage which should increase throughput down the road, and we will continue to send limited bursts of ARP1 work to establish what the consistent rate will be in the fall.
- We await final confirmation from TU Delft that the thousands of results we have validated and shipped to them from the new BOINC backend so far are consistent with earlier results and we should keep them coming. We have no reason to believe otherwise, but this must be confirmed explicitly and we expect movement on this by mid-September, 2026, if not before. Until then we will continue to distribute workunits and update volunteers. The project remains scientifically and technically relevant based on discussions with TU Delf, and we are exploring plans to expand climate research at WCG using the ARP1 model.
- The ARP1 specific files generations.txt, state.txt, and completed.txt will be published again when the tasks mentioned above are further along. We anticipate mid-September, after we review with TU Delft, as the earliest reasonable date. We could restart the script as is, and it would run against the legacy database as the stats dump currently does, but we would rather restructure it to connect directly to the new BOINC database while producing files in the exact same format so we can avoid more synchronization issues.
- ARP1 extremes from older generations were reviewed, some restarted successfully, some still have issues that must be resolved before they can be distributed. Extremes and BOINC client errors that have now been masked by the transitioner as having too many errors to continue distributing resends are being assessed. When we are able to restore the ARP1 specific stats export as mentioned in the previous bullet point, progress or lack thereof on these will become visible as before.
MAM1
- We have begun testing a ROCm AMD GPU build for MAM1 on Linux in the beta30 project.
- Updated versions of the Windows and Linux NVIDIA CUDA GPU builds and multi-threaded CPU builds for the Mapping Arthritis Markers (MAM1) beta30 project have been released, and will be promoted to production MAM1 next week. We plan to release and maintain Windows, Mac, Linux, Android on AMD, Intel, Arm, NVIDIA, Apple Silicon and Metal over time.
- We have since made significant changes to MAM1 over the summer of 2026, improving the neural network architecture, changing memory management on the GPU to provide a guard against OOM crashes on volunteer devices, changing the optimizer from Simulated Annealing to Differential Evolution to fix the low GPU utilization which previously remained even after changing the cross validation logic to train multiple folds simultaneously. Other bugs as they were discovered or reported on the forum were fixed, and there will likely be more as we continue the rollout.
- All recently issued work under the beta30 and MAM1 project has been viable science for some time. Gene signatures are evaluated between batches to plan and configure subsequent batches. We have implemented and tested an automated procedure for gathering promising genes and subsets of genes from the data lake where signatures from previous batches are stored, and minting new batches based on the best performing signatures we have. We will continue to build up this infrastructure with the expectation it will be applied to MCM1 and other diseases going forward. For this reason, occasional Ovarian and Lung batches have been released and will continue to be released as MAM1 beta30 work. MAM1 gives us MCM1 on GPU, which we plan to pursue soon.
- We are working to resolve packaging issues for the DLLs published with some of our Windows releases for beta30, which has recurred multiple times as we iterate on the beta30 application versions. Although we are able to fix these problems with subsequent releases each time, this has happened multiple times in the beta30 release lineage. Some beta30 applications were also promoted to a MAM1 application version before we identified the issue with the beta30 bundle. The issue continues to affect some volunteers, often showing up as runtime errors, and we are working on a more formal release process to ensure this doesn't happen again.
- We introduced a regression to multi-threaded CPU-only MAM1, now fixed. OpenMP and LibTorch should now respect BOINC --nthreads again, thank you to volunteers who reported this in the forum. Appears to have been caused by an added include of torch.h in a CPP source file where it should not have been.
MCM1
- MCM1 beta31 project that runs on NVIDIA/AMD GPUs will be added, soon. However, we must do this in a way that allows currently participating devices to continue to have an impact as they process MCM1, while benefiting from the increased capabilities of the MAM1 GPU capable platform run on the MCM1 datasets. What we determine here will also be the path toward allowing less capable devices to run MAM1 productively. Likely, the backends other than LibTorch, dlib and OpenCV, will be used to train and evaluate SVM and RandomForest classifiers, with the same OpenMP based heuristic search between signature or signature population evaluations provided by our serializable build of the ensmallen library. While this will require a bit of development to ensure those backends still behave well with all the changes we have made while prioritizing the LibTorch backend, these workunits would run on older machines and support followup investigation or rapid hypothesis testing for the most promising gene signatures first identified by MCM1 with LibTorch. While the neural network architecture developed for MAM1 already works for MCM1, there must still be a separate analysis and development phase before MCM1 on GPU is ready to release within the MCM1 application lineage. However, the foundation has been laid with MAM1 application development, and the implementation of the data lake supported by the postgres DuckDB plugin ecosystem.
- Validation for new work MCM1 work is finally healthy as volunteers have reported in the forum, but we continue to recover pending validations.
Updates on the status of the Africa Rainfall Project (ARP1), Mapping Arthritis Markers (MAM1, beta30), and Mapping Cancer Markers (MCM1) projects at WCG. Updates on web and API development that is currently in progress will be provided next week.
ARP1
- ARP1 workunits are being distributed again, consider the project intermittent pending a full restart. ARP1 workunits are orders of magnitude larger than MAM1 and MCM1 workunits, so the in-memory cache across a partitioned BOINC backend that allowed multiple backend servers to share the load, and work around the slower NFS performance and database IO at Nibi compared to the dedicated appliance we had at Graham, is simply less beneficial for ARP1. Further, it negatively impacts MCM1 and MAM1 for little benefit, as we simply cannot hold enough ARP1 workunits resident in memory to make a dent in throughput anyways. We now have a working ARP1 lifecycle based on our NFS and block storage as we await access to object storage which should increase throughput down the road, and we will continue to send limited bursts of ARP1 work to establish what the consistent rate will be in the fall.
- We await final confirmation from TU Delft that the thousands of results we have validated and shipped to them from the new BOINC backend so far are consistent with earlier results and we should keep them coming. We have no reason to believe otherwise, but this must be confirmed explicitly and we expect movement on this by mid-September, 2026, if not before. Until then we will continue to distribute workunits and update volunteers. The project remains scientifically and technically relevant based on discussions with TU Delf, and we are exploring plans to expand climate research at WCG using the ARP1 model.
- The ARP1 specific files generations.txt, state.txt, and completed.txt will be published again when the tasks mentioned above are further along. We anticipate mid-September, after we review with TU Delft, as the earliest reasonable date. We could restart the script as is, and it would run against the legacy database as the stats dump currently does, but we would rather restructure it to connect directly to the new BOINC database while producing files in the exact same format so we can avoid more synchronization issues.
- ARP1 extremes from older generations were reviewed, some restarted successfully, some still have issues that must be resolved before they can be distributed. Extremes and BOINC client errors that have now been masked by the transitioner as having too many errors to continue distributing resends are being assessed. When we are able to restore the ARP1 specific stats export as mentioned in the previous bullet point, progress or lack thereof on these will become visible as before.
MAM1
- We have begun testing a ROCm AMD GPU build for MAM1 on Linux in the beta30 project.
- Updated versions of the Windows and Linux NVIDIA CUDA GPU builds and multi-threaded CPU builds for the Mapping Arthritis Markers (MAM1) beta30 project have been released, and will be promoted to production MAM1 next week. We plan to release and maintain Windows, Mac, Linux, Android on AMD, Intel, Arm, NVIDIA, Apple Silicon and Metal over time.
- We have since made significant changes to MAM1 over the summer of 2026, improving the neural network architecture, changing memory management on the GPU to provide a guard against OOM crashes on volunteer devices, changing the optimizer from Simulated Annealing to Differential Evolution to fix the low GPU utilization which previously remained even after changing the cross validation logic to train multiple folds simultaneously. Other bugs as they were discovered or reported on the forum were fixed, and there will likely be more as we continue the rollout.
- All recently issued work under the beta30 and MAM1 project has been viable science for some time. Gene signatures are evaluated between batches to plan and configure subsequent batches. We have implemented and tested an automated procedure for gathering promising genes and subsets of genes from the data lake where signatures from previous batches are stored, and minting new batches based on the best performing signatures we have. We will continue to build up this infrastructure with the expectation it will be applied to MCM1 and other diseases going forward. For this reason, occasional Ovarian and Lung batches have been released and will continue to be released as MAM1 beta30 work. MAM1 gives us MCM1 on GPU, which we plan to pursue soon.
- We are working to resolve packaging issues for the DLLs published with some of our Windows releases for beta30, which has recurred multiple times as we iterate on the beta30 application versions. Although we are able to fix these problems with subsequent releases each time, this has happened multiple times in the beta30 release lineage. Some beta30 applications were also promoted to a MAM1 application version before we identified the issue with the beta30 bundle. The issue continues to affect some volunteers, often showing up as runtime errors, and we are working on a more formal release process to ensure this doesn't happen again.
- We introduced a regression to multi-threaded CPU-only MAM1, now fixed. OpenMP and LibTorch should now respect BOINC --nthreads again, thank you to volunteers who reported this in the forum. Appears to have been caused by an added include of torch.h in a CPP source file where it should not have been.
MCM1
- MCM1 beta31 project that runs on NVIDIA/AMD GPUs will be added, soon. However, we must do this in a way that allows currently participating devices to continue to have an impact as they process MCM1, while benefiting from the increased capabilities of the MAM1 GPU capable platform run on the MCM1 datasets. What we determine here will also be the path toward allowing less capable devices to run MAM1 productively. Likely, the backends other than LibTorch, dlib and OpenCV, will be used to train and evaluate SVM and RandomForest classifiers, with the same OpenMP based heuristic search between signature or signature population evaluations provided by our serializable build of the ensmallen library. While this will require a bit of development to ensure those backends still behave well with all the changes we have made while prioritizing the LibTorch backend, these workunits would run on older machines and support followup investigation or rapid hypothesis testing for the most promising gene signatures first identified by MCM1 with LibTorch. While the neural network architecture developed for MAM1 already works for MCM1, there must still be a separate analysis and development phase before MCM1 on GPU is ready to release within the MCM1 application lineage. However, the foundation has been laid with MAM1 application development, and the implementation of the data lake supported by the postgres DuckDB plugin ecosystem.
- Validation for new work MCM1 work is finally healthy as volunteers have reported in the forum, but we continue to recover pending validations.