The researchers designed six benchmarks to assess the models' spatial reasoning and logic skills. These tests included tasks such as the "Barman" cocktail preparation challenge, the "Blocks World" stacking puzzle, and the "Floor Tile" painting problem, among others. Each task was designed to be easily solvable by humans but traditionally difficult for AI.

Key findings from the study include:

1. Improved Rule Understanding: The O1 models, particularly O1 Preview, demonstrated a superior ability to understand and follow complex rules compared to GPT-4. This improvement is attributed to O1's capacity for "test-time compute," allowing it to reflect and plan during inference.