Benchmark Domains and Problems of IPC-5

For IPC-5 we designed five new planning domains, together with a large collection of benchmark problems. Moreover we used two domains from previous competitions for which we developed some new variants. The new IPC-5 test domains have different motivations. Some of them were inspired by real world applications, (e.g., storage, trucks and pathways); others were aimed at exploring the applicability and effectiveness of automated planning for new applications (pathways), or for known problems that have been addressed in other fields of computer science (TPP and openstacks); finally, two domains were taken from previous competitions, as sample references for the advancement of automated planning with respect to the existing benchmarks (rovers and pipesworld).

For some domains, the problems we generated have many solutions. In these problems, the most challenging aspect is finding plans of good quality, which is the main issues on which this competition focused. Other problems are challenging for different reasons: the expressiveness of the planning language used to model the problem including some of the new features of PDDL3.0, the large size of the problem, or the known NP-hardness of the computational problem they model. In most cases, the test problems were automatically (or semi-automatically) generated by using dedicated software tools.

In order to make the new PDDL3.0 language more accessible to the IPC-5 competitors, for each domain we developed several variants, using different fragments of PDDL3.0. The "propositional" and "metric-time" variants use only the constructs of PDDL2.2; the "simple preferences" variant extends the propositional with preferences over the problem goals; the "qualitative preferences" variant also includes preferences over state trajectory constraints; the "metric-time constraints" variant extends the metric-time variant with strong state trajectory constraints; and, finally, the "complex preferences" variant uses the full power of the language, including soft trajectory constraints and goals. However, not all the different variants of each domain actually use the full fragment "allowed" for that variant. In addition, for some domain variants we provided additional equivalent versions were conditional effects or universally quantified variables in PDDL3 constructs are compiled away.

In the domain variants involving preferences we created for each planning problem a plan metric incorporating terms specifying the penalties for violations of the preference. The metric is a very important part of the problem statements in such domains, since it determines which is the best trade-off between different, possibly mutually exclusive preferences, and we tried with much care to ensure that the metrics in the test problems give rise to challenging optimization problems.

An archive with all the benchmarks files is available here, while the problem generators for the competition domains are available here. Below we list the domains of IPC-5 and give a short description of them.