(1) Explain your problem, don't simply post "This isn't working". What were you doing when you faced the problem? What have you tried to resolve - did you look for a solution using "Search" ? Has it happened just once or several times?
(2) It's also good to get feedback when a solution is found, return to the original post to explain how it was resolved so that more people can also use the results.
Dear all,
I have been using MS Project for a while and now have allocated some time to look at other programs. As a part of this investigation I compared resource leveling algorithms of them (I am one of those, who believe, that resource leveling is very important for the scheduling software).
So, I just want to share my results. I also welcome your comments. As this is related to several tools, I am posing this here in a none-tool specific part of the forum.
Tools compared:
Tests conducted
I have done resource leveling on the following 2 schedules
1) Intuitive simple 4 tasks and 2 resources schedule, which I call for myself “Vladimir Liberzon Leveling Test”.
You can see the schedule here:
2) Schedule j305_test5.001, which Iwas also kindly created for me by Vladimir Liberzon, based on one of the problems from the library of Resource-Constrained Project Scheduling Problems (RCPSP), made available by Technical University of Munchen.
For this schedule I noticed, that Primavera produces different results, depending on the sequence of events (depending on which Activity ID has been chosen for a task). Therefore to make a correct comparison I have also done testing with the other programs using 2 different sequence of activities.
You can get this schedule in formats of Spider Project, MS Project and Primavera XER file
How testing was done:
I would run standard resource leveling without any priorities. So effectively the tool was supposed to find the shortest possible schedule, without looking at task priorities.
For “Vladimir Liberzon Leveling Test”, if the tool would produce human-like result, then the test would Pass, otherwise it would fail. (Human can easily see, that to avoid resource overloading, Task3 shall be delayed by a day.)
Results:
Results are presented in the table below:
Note: for test j305_test5, the results are shwon in calendar days.
| Spider Project (optimisation plus mode) | Spider Project (standard mode) | Asta Powerproject | Primavera P6 R8.2 | Microsoft Project 2007 | Microsoft Project 2010 | |
|---|---|---|---|---|---|---|
| "Vladimir Liberzon Leveling test" | Pass | Fail | Fail | Pass | Fail | Fail |
| Schedule j305_test5 (1st sequence of actions) | 54 | 60 | Not tested (schedule does not fit demo version of Powerproject) | 62 | 60 | 65 |
| Schedule j305_test5 (2nd sequence of actions) | 54 | 60 | Not tested (schedule does not fit demo version of Powerproject) | 76 | 60 | 65 |
Conclusions & observations:
Appendixes
MS Project 2007 results screenshot
Primavera results screenshots
Sequence 1
Sequence 2
Spider results (optimization plus mode).
Printer-friendly version
Send to friend
This discussion is now followed in another thread: http://www.planningplanet.com/forums/planning-scheduling-programming-discussion/574396/comparison-resource-levelling-algorithms-di
Jack,
I just checked, and the file j305_test5_1.mpp is still available in the link
Regards.
Hi All. Is it possible to get J30 set in mpp. It is already not available. I would like to test it againts Pritsker algorithm. Thanks.
Vladimir,
1 year ago you have written "..They had sent to me test results and I was informed that revised paper will be published but it looks like it was not published yet.."
Just wondering, has it been published now?
Regards.
Evgeny
Ken,
thanks for suggesting a new product to test.
I must say, I have done the testing, described in this topic last Christmas, when I had a bit of free time. Now I am not sure, when I will have this luxury again.
Why don't you run your tool on the schedules, which I used for my testing, and post results here with description, detailed enough, so that people can reproduce the results? Would be very interesting to see how your results compare with others.
I looked at the FAQs of the MSP_LEVELING. I can see, that you use MS Project end enrich it with extra resource leveling options (some of which pretty cool and I don't think are present in most of the tools, e.g. Substitutions ).
You probably have done market research, so this won't be new for you, but based on my testing as well as the feedback from people on this forum, Spider Project is a benchmark for resource leveling so far. It feels like, it has pretty much all the resource leveling features, which are described in FAQs of MSP_LEVELING, avoids limitations of it, plus more resource leveling / modelling /optimisation features.
And, when playing by the rules of MS Project and Primavera (not involving advanced features, only basic leveling), it also wins, as you can see from this topic. But may be MSP_LEVELING has algorithm, which can beat Spider?
Regards.
Evgeny
Hi Evegny
I just became aware of your post - interesting results.
Are you able/willing to test another leveling engine?
The leveling engine software in question is brand-new.
We, at ARA, call it MSP_LEVELING. It levels MS Project schedules (and more).
For more information, see the website: ara-project.com.
Please note that the website is still being developed (a work in progress), so it may not be really clean (or complete) at this time.
You'll get the best information by reading the FAQs page.
Status:
I'm now in the tail-end of Beta testing (a week or two left), so - if you're interested time is kinda short (sorry).
I'm willing to send you, and any Planning Planet member that you recommend, a full-capability registered copy of the software.
Information required: your Name and Email address.
The only qualifier is that I ask those who get the Beta test to give me feedback (likes/dislikes/test results) no later than the first week in December (about two weeks from now at latest).
Please feel free to contact me.
Best regards,
Ken
Ken Vidacovich
Advanced Resource Analytics Inc
504-333-9068
Q1. The article is quire recent (Y2009), Spider probably was at the market back then (Also Powerproject). Why was it not included the in the study?
We never marketed Spider Project and it became popular just because people were happy with its capabilities and told others. Now it is most popular in Russia and is used in most large scale programs but still is not widely known abroad. In 2010 they wrote to me that tried Spider Demo with J30 set and were surprised by the results that were the best. I supplied them with more powerful version to test J60 and J120 sets where Spider also showed the best results. They had sent to me test results and I was informed that revised paper will be published but it looks like it was not published yet.
Q2.The paper says, that they were testing setting different priority rules in tools “We have tested 3150 priority rule combinations in CSP, and 196 priority rule combinations in PP6. In ACO, a single-level rule is applied; here, the user can choose between 8 priority rules. We have tested all of them.”
Yes, trying many prioritiy rules is time consuming. Besides, simple priorities used by most packages do not guarantee that the result will be close to optimal. Spider Project uses iterative algorithms that are more sophisticated. But still an ability to use different schedule priorities is better than being restricted by only one standard leveling heuristics like in MS Project.
Q3 Re: “The results indicate that none of the tested software packages is currently competitive with the best state-ofthe-art algorithms from the literature.”
I suspect that state of the art algorithms from the literature require a lot of time for schedule calculation. They can be applied to 30 or 60 activity schedules but even for 120 activities set not all optimal results were known. We deal with the schedules consisting of many thousands (for portfolios hundreds of thousands) activities. So we need to find a balance between good enough optimization and time it may take. We can make our algorithms even more powerful but it will meant an increase of calculation time.
Besides most of these state of the art algorithms were developed for simple schedules that use FS dependencies and full time resource assignments. With variable resource assignments, teams, double dependencies, etc. they do not work. Spider algorithms were not taken from the literature.
As I wrote earlier absolute optimums may be found by trying all possible combinations. For J30 and J60 sets they are known, for many test projects of J120 set absolute optimums are not known yet. For projects consisting of thousands activities computation time for finding absolute optimums is too large and approximation is necessary.
P.S. Did you try http://rapidshare.com/files/3936006972/j309_9example_1.mpp with MS Project 2007 and P6?
Best Regards,
Vladimir
Rafael,
Thanks for the link for publications. As always, you came up with very interesting link. I must say, I spent some time reading it, as well as some other (though not all) documents, located here:
http://webserver.wi.tum.de/psplib/library.html
Vladimir,
I just have a couple of questions regarding this article.
Q1. The article is quire recent (Y2009), Spider probably was at the market back then (Also Powerproject). Why was it not included the in the study?
Q2. The paper says, that they were testing setting different priority rules in tools “We have tested 3150 priority rule combinations in CSP, and 196 priority rule combinations in PP6. In ACO, a single-level rule is applied; here, the user can choose between 8 priority rules. We have tested all of them.”
In my really humble opinion, for the purpose of their testing the software shall not have all of these thousand priorities, also it does not make too much sense to test their effects on the results. There should be just one setting: “I want the shortest schedule!” (pretty much, like in Spider) Making user try to improve results by playing with them is close to manual resource leveling.
Or I am missing something?
Q3 Re: “The results indicate that none of the tested software packages is currently competitive with the best state-ofthe-art algorithms from the literature.”
How can it happen in this competitive word? If the algorithm is available in the literature, why was it not implemented in the tools, studied?
Does it mean, that Spider has implemented these “state-of the-art algorithms from the literature”
Separately:
On Sun, 2012-12-30 you have written: "......Numbers show percent of delay if to compare with absolute optimums....."
Question: who, how and based on which algorithm has calculated the absolute optimum?
Regards.
Evgeny
Evgeny,
different industries have different requirements but designing most simple and cheap Spider Project Lite version we included there activity volumes and resource productivities, resource constrained schedule optimization and skill scheduling as features that are "must have". Another "must" feature is reference-books that are necessary for setting corporate standards and avoid errors and multiple entry of the same data. Pay attention to this most powerful feature that is most important.
In Telecom industry projects rarely can be managed separately and portfolio management is necessary. By portfolio management I mean among other features an ability to calculate portfolio schedule taking into account all existing constraints. But this is Spider Project Professional feature.
Regards,
Vladimir
Evgeny,
I believe they all have some functionality for skills, but I expect it might be very limited. I doubt they would implement skills with different productivity so that when a lower productivity resource is selected then the activity duration is increased.
This might be another challenge to resource leveling as you want the software to assign skills in a way total project duration is minimum. Some rule like amount of resource hours in descending order might be needed, or maybe other rule(s) in order for the software decide on which of the competing activities to assign the more productive skill. The more functionality you add the more challenging and more difficult resource leveling becomes, not to mention manual resource leveling. Unfortunately if these software are incapable of handling skills with different productivity we will not be able to compare them with Spider.
In our construction jobs most activity durations are driven by resource productivity, not being able to model resource productivity is a serious handicap. Some labor resources are more productive than others as well as equipment, it is common to use some equipment with lower productivity when the higher productivity is assigned to other activity. The effect of this assignment must be considered, not ignored.
Best regards,
Rafael
Vladimir,
correct me if I am wrong, but I think what is important to mention, is that automatic resource leveling becomes much, much more attractive, when used with Automatic Resource Assignment Based on Skills feature of Spider (was not tested in this report, as no other tool supports it, as far as I know). This is when you assign not resources, but skills to a task. At the same time every real resource may have several skills assigned to it. And then the tool will walk through the schedule and assign the real resources to achieve the most optimal schedule and also would resource level it at the same time
This seems to be an extremely powerful feature for me. I work in telecom, and we don’t have so big schedules as in building industry, however what makes a life difficult is constantly changing situation: resources get removed, another, with different skills get added, new projects emerge, another ones get reprioritized, issues happen, which need to be addressed etc). In addition to this, there are constant “what if” questions from management. E.g.: what will happen with the program if we remove this resource? What will happen if we add this project? Without this Skills feature and without automatic resource leveling, which I can trust, it takes me often days in Microsoft Project to answer a single “what if” question. So, needless to say, that I just don’t have these extra days in my possession and no proper analysis gets done often.
You challenge people here to do a manual resource leveling and to compare it with automatic. But I wonder, if you can challenge them also with a schedule which has skills (not resources), assigned to tasks and then separately with resource skills matrix and then also with separate resource productivity for every resource for every task. I wonder how humans will even approach this situation.
Regards.
Evgeny
Rafael,
they had sent to me Spider test results together with test resulys of other packages but did not inform if the paper was published and did not answer to my recent inquiry. Maybe this is censorship - I don't know.
The results that I published here are only a part of their tests that included also testing with J60 and J120 sets (where Spider also showed the best results). With higher dimensions the results became worse - that is why you remember worse numbers. But I have no rights to disclosure their data without their permission, so I cited only specific J30 data because sample projects that I uploaded belong to J30 set.
I selected only those projects where Spider results are not optimal for a fair play. But it looks like they are still the best. Opposite to Lord assumption I tried to give a chance to our competitors and to manual leveling to prove that they may produce better results for small schedules with FS links and full time resource assignments.
Best Regards,
Vladimir
Vladimir,
I just noticed your reference to Berne University and wonder if they ever published a comparison that includes Spider as for years I have seen their comparisons and Spider was never present. I was under the impression they yielded to censorship.
Is the report that includes Spider available or still in the works?
They studied each software and used the best solution they found based on several suggested prioritization rules for each software, something many schedulers are unaware.
The percentages are a surprise to me because as far as I recall on previous publications in which Spider was not included the results were much worse than 10% from best known solution, not to be confused against unleveled schedule. I believe in one of their studies they said MSP was as bad as any other and suggested a somewhat complicated procedure to get as bad solution instead of worse than bad.
To me 2% deviation by Spider from best known solution is awesome, especially if at a single click, without playing with a multiple prioritization rules. I am still trying to figure out about the new percentages and maybe you can give me some light.
http://dde.teilar.gr/sectionAttachments/215/trautmann[2010a].pdf
They said - We have tested 3150 priority rule combinations in CSP, and 196 priority rule combinations in PP6.
Is it practical?
Best regards,
Rafael
Peter,
an information required for automatic and manual leveling is the same.
Manual leveling can be done in small and simple projects but when there are many activities competing for limited resources it is too time consuming and not reliable.
Resource leveling is done during the project life cycle after any input of new information. Your initial "false" links may become false constraints very soon if you do not reconsider them regularly.
Automatic leveling is fast, reliable and saves a lot of planner's time and effort providing fast answers to what if questions. And optimized project schedule saves a lot of money for the customer.
You can try to level manually small 30 activities project suggested to Mike earlier in this discussion and compare with the automatic levelling results. Project file was upoaded in MSP 2007 format. If you want to try but need some other file format let me know.
To the moment two results of this project leveling were published: MS Project 2013 duration 85 days, PowerProject duration 76 days. Manual leveling duration - ?
Best Regards,
Vladimir
Even if you do not perform a resource leveling run resource loading still makes much sense sense, it takes not much time if you know how to. It is good practice and frequently required by contract as well as by some courts that have established as common law that resources must be considered on the schedules or otherwise might be considered not valid for any claim, no matter if you like it or not.
It does not make sense not to resource load your schedule even if you do not perform any resource leveling after putting some effort into your schedule as the software will warn you of over allocations at a single click. Why being so afraid of disclosing over allocations and pretends they will be solved by magic?
At times resource leveling will tell you got to plan for overtime, for a second shift, for more productive resources; it is not always about the best algorithm. If you do not adequately resource load your schedule there will be no warning.
Schedules are dynamic, a resource that at a time is not critical latter on might be. Not considering relevant and near relevant resources is like leaving relevant and near relevant activities out of your schedule, resources and activities go hand to hand, without resources you will get nothing done, not planning for resources is wrong practice.
Even if your software is so poor as for you to consider not using its resource leveling capabilities you will still have to plan and use watever resources it takes to get the work done.
Hi All
I agree with Mike Testo - I wouldnt trust any software to do resource levelling as the amount of time you would need to spend to make sure that the software had all the info and data required to make the call, it would be easier and quicker to manually move a few tasks around using lags or 'false' links to ensure optimisation of the prime resources, and then letting the non-essentials work themselves out.
Happy New Year
Peter Hibberd
Thank you Mike!
My best wishes to you in 2013!
We always level resources using Spider Project and people expertise. Together they work perfect.
Best Regards,
Vladimir
Hi Vladimir
It was the default PowerProject setting per Day granularity.
The last time I used resource levelling on a live project was with the Road Maintenance programme for Suffolk County Councill.
They only had two tarmac machines (Barber Greene's) for the whole county road network so it was useful to avoid arguments from the four depots that needed the kit.
They seemed to accept that the computer had allocated the resource rather than the County Engineer - although he was the man behind the priorities.
Wishing you a prosperous and content 2013.
Best regards
Mike T.
Mike,
machine helps to make a decision.
Considering and evaluating the schedule suggested by the software you can set manual priorities and estimate "what if" results.
Manual leveling is possible in most simple cases and still it shall be compared with the schedules suggested by the software that may suggest better decisions. Project Management software is a tool that helps to manage projects but it will never replace a person that makes decisions taking into account soft factors.
Thank you for PowerProject result. It is much better than produced by MS Project 2013 (85 days if to level hour by hour and 86 days when to select minute by minute). Is it default PowerProject schedule or you tried different options and selected the best leveling priority? If yes, what parameter was selected for the best schedule?
Best Regards,
Vladimir
Hi Vladimir
The calendar is Nr 5 24hrs 7 days - Asta set the start date to today 31st Decemebr.
The levelled duration is 76 calandar days 0 Hours.
I accept that cross project requirements for specific plant items or identified skilled labour can be controlled and allocated by resource levelling software - but it is still the machine that is making your decisions.
Best regards
Mike Testro
Mike,
In addition our real life assignments, no matter if a dummy software/scheduler cannot model it, at the jobsite we frequently assign resources to the same activity working on different shifts. Fortunately this is easy to model with Spider while many other software have problems with correct modeling of shifts on a same activity. Usually 2 labor crews work on different shifts while equipment is shared. The progress of an activity, the finish of an activity is dependent on the work on all shifts.
This exponentially complicates manual resource leveling as for the day shift the demand of equipment is usually more than during the night, this does not means one shift will stop the other but perhaps during some shortage of equipment depending on priorities some day shift will not be able to work while night shift will help to do the work on the same activity.
Obviously the use of hammocks to model limited equipment availability is useless.
We model these activities as production type so the availability and productivity of resources will tell us when work will be done and finished. Usually productivity at night shift is lower but it might happen otherwise.
When I was a SureTrak user it was common for us to use it to comply with a contractual requirement and set it aside to do actual resource management due to the lack of functionality. Today Spider gives us better results than trying to do it manually because for most simple of jobs resource leveling is still too complicated.
Best regards,
Rafael
Hi Mike,
This example is not mine – it is one of many test projects from J30 set created in 1995 for testing resource constrained scheduling algorithms.
In the live projects we assigned crews that include many resources of different types. And the same resources are required at different jobs competing for them. Mostly these limited resources are machines (excavators, bulldozers, cranes, etc.) but sometimes they are skilled workers that are not easy to hire at any moment and to fire when they are not needed.
Project schedule that I had sent starts on 20 September 2010. You (or PowerProject) obviously selected different date and it does not matter. What is overall duration in days and hours? I am not sure that a calendar will be imported properly, so duration in hours may confirm it.
Hi Vladimir
Your example with multiple resourses under one task was impossible to level using soft links.
It could have been done by creating hammocks of single resource tasks but that would have taken too long and would have been pointless anyway.
I know it was an hypothetical example but would you really set up such a programme on a live project?
PowerProject resource levelling took 5 seconds and ended at 16th March 2013.
Hi Vladimir
I never looked at any of the results anyway.
Best regards
Mike Testro
Mike,
After knowing the answer anyone can trick the software to get it right by creating artificial links [or date constraints]. In order to prove you are getting it right by yourself you must work a job where the answer was not disclosed.
Best regards,
Rafael
Thank you Mike!
Remember that there are 12 units of resource A, 15 units of resources B and C, and 16 units of resource D.
I am interested in overall project duration calculated by Asta and improved by yourself.
Best Regards,
Vladimir
Hi Vladimir
Thanks for that - I have it open in PowerProject and I will start in the morning.
I assume you wish for resources A B C D to be levelled so that they do not overlap.
I will let you know the levelled end date and I will time the excercise.
Best regards
Mike Testro
Mike,
another example from J30 set was uploaded to http://rapidshare.com/files/3936006972/j309_9example_1.mpp
It is MSP 2007 file and I expect that Asta will be able to open it.
I did not prepare these sample projects as Lord supposed, these examples were created in 1995 and include only FS links, full time resource assignments, nothing complicated like shifts, partial and variable assignments, skills, etc.
Calendar is 24 hours, but in any case durations are set in hours.
So nothing complicated, just pure classic scheduling.
Best Regards,
Vladimir
Hi Vladimir
If there were 4 resources that needed to be levelled manually in Asta PowerProject I would say the exercise would take about 10 minutes.
5 minutes to set up the 4 link categories and another five to filter on each resource laden activity and link them up with the Select All > FS Link Icon.
The resource links can then be swithched on or off as you wish.
Best regards
Mike Testro
Censorship by Software giants have effectively stopped benchmarking even if no chance in court.
I doubt you will find serious and independent Benchmarking that compares Spider to these giants, they allowed benchmarking before real competition surfaced.
I believe software that prohibits benchmarking shall be ruled out for use by our government as there is no independent way to justify its use.
Lord,
of cause these results are indicative.
I used tests, which I got from Vladimir, just because these were the only ones available to me. Vladimir provided them originally to test different Spider leveling algorithms.
Still, I believe these results quite informative because of the following:
One can not really suspect, that Vladimir is biased towards MS Project against Primavera or towards MS Project 2007 against MS Project 2010.
Nevertheless I take your point. I will try to contact representatives of other manufactures to get their suggested benchmark schedules.
I will post progress here.
Regards.
Evgeny.
Hi Lord,
J30 is a set of standard tests not developed by me. There are a lot of them and I can give a reference to the site where they are placed together with J60 and J120 series. These tests were used in Operation Research and a group in Berne University recently tested many packages with all of them (many of these test projects are very simple and all packages produce close or the same results, others complicated like two examples I suggested to Evgeny in other thread).
Overall results for testing J30 set (several hundreds sample projects) default leveling algorithms were as follows:
MSP 5.18
PS8 4.93
P6 9.45
Spider 2.24
Numbers show percent of delay if to compare with absolute optimums. MSP means Project 2007. To my great surprise Project 2010 and 2013 produce much worse results with the same tests. They also tested other packages that are not presented here in PP but results were not better.
If you will want to repeat the tests yourself I will give you a reference.
I would like to get the test where MS Project or Primavera produce better schedule than Spider. It will help us to improve Spider capabilities. Let me know if you will meet such example.
Best Regards,
Vladimir
I was also pretty impressed with the tests.
Until I found out that all your tests were designed by Vladmir.
Nothing personal, but I feel tests designed by one of the 'comparision software' makers, and results thereof is not really an 'indicative' result.
I think if you ask Microsoft, they will provide a 'special schedule' that, when 'resource levelled' will be the best compared to all. Even Spider.
Same with Primavera, Asta and the lot.
Is it possible? I mean, can a project software developer provide a schedule such that when it runs in 'his project software' will provide the best resource levelling? - I don't know.
But the results are pretty colored to me. If they are not, I apologize to Vladmir and everyone here (it 'could' be the correct results also!), but this report comparing a feature of a project software, with a test 'assisted' by one of the guy who designed one of the software in comparision -- I cannot seem to think it's neutral.
Sorry. I'll give THIS comparision a pass.
Nice effort, though.
Best Regards,
L_H
Mike,
what if you will level manually a project that consists of only 30 tasks like one used by Evgeny in his test and we will compare manual schedule with the schedule that Spider will create automatically?
It could be an interesting test for software capabilities.
If you agree I will publish a test project here in MS Project format or just as a text description. And don't forget to estimate what time your manual leveling will take?
Best Regards,
Vladimir
Hi Evegny
Well done.
Your splendid efforts have proven that Resource Levelling may produce a result but not necessarily the one that is best for your project.
Resource levelling is often necessary for scarce resouces but I would never allow the software to make such crucial decisions for me based on a set of abstract priorities.
Keep up the good work
Best regards
Mike Testro
Email: PPadmin@planningplanet.com
Replies